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Foreword 
By the Editors of the Series 


In the field of psychology we believe that the student 
ought to get the “feel” of experimentation by reading 
original source materials. In this way he can acquire a 
better understanding of the discipline by seeing scientific 
ideas grow and change. However, one of the main prob- 
lems in teaching is the limited availability of these sources, 
which communicate most effectively the personality of 
the author and the excitement of ongoing research. 

For these reasons we have decided to edit several 
books,* each devoted to a particular problem in psychol- 
ogy. In every case we attempt to select problems that have 
been and are controversial—that have been and still are 
alive. We intend to present these problems as a set of 
selected original articles, arranged in historical order and 
in order of progress in the field. We believe that it is 
important for the student to see that theories and re- 
searches build on what has gone before; that one study 
leads to another, that theory leads to research and then 
to revision of theory. We believe that telling the student 
this does not make the same kind of impression as letting 
him see it happen in actuality. The idea is for the student 
to read and build ideas for himself. 

Suggestions for Use—These readings books can be used 
by the student in either of two ways. They are organized 
so that, with the help of the instructor (or of the students 
if used in seminars), a topic can be covered at length and 
in depth. This would necessitate lecturés or discussions on 
articles not covered in the series to fill in the gaps. On the 
other hand, each book taken alone will give a student a 
good idea of the problem being covered and its historical 


background as well as its present state and the direction 


it seems to be taking. 


* (Pub. note: a sub-series within the Insight Book Series.) 
iii 
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Preface 


The field of programmed learning, or auto-instructional 
materials (erroneously called “teaching machines”), is a 
rapidly developing, extremely exciting movement toward 
the further development of an instructional technology 
based, in part, on certain principles of psychology derived 
from laboratory studies of human and animal learning 
which serve to confirm, in part, the procedures and tech- 
niques used by nearly all truly great teachers of the past. 

Many professions other than psychology have a legiti- 
mate interest in instructional technology, e.g., specialists 
in the fields of audio-visual materials, curriculum develop- 
ment, engineering, systems development, teacher educa- 
tion, educational administration, and communications. 
This set of readings concerns itself with only a small part 
of the field of instructional technology, programmed 
learning, and it stresses that material of greatest interest 
to psychologists, particularly educational psychologists. 
Only in the first article, Finn’s “A New Theory of Instruc- 
tional Technology,” will the reader find the broad concern 
with this topic. Finn’s article represents, then, the broad 
context within which one might view some of the work of 
several professions, including psychology. 

Psychology’s concern with an instructional technology 
is rather recent, and it has been limited chiefly to evalu- 
ative studies of media. Its interest in programmed learn- 
ing and auto-instructional devices is far greater than its 
interest in other media, since two psychologists, B. F. 
Skinner and Sidney L. Pressey, have been the pioneers in 
the development of self-teaching devices and materials. 
Beginning with Skinner's 1954 paper entitled “The Sci- 
ence of Learning and the Art of Teaching,” psychology’s 
claim to a contribution to developing an instructional 
technology gained considerable support. In a sense, Skin- 
ner proposed that laboratory studies of animal learning 
serve as a base for a science of teaching; however, earlier 
work with human learning is not to be considered irrele- 
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vant to the.developments in programming. The work of 
Ebbinghaus and his followers, of Thorndike and his stu- 
dents and of Pressey and his students inevitably bears a 
relationship to programmed learning. b 

A casual perusal by a humanist of the rapidly growing 
literature on self-instructional devices and programmed 
material would probably give rise to either the fantasies 
of Alice in Wonderland and the Wizard of Oz or a vision 
of Brave New World, Walden Two and 1984. These 
views of the field, however, are quite inaccurate, as one 
can discover by critical reading of those articles prepared 
within a scholarly tradition rather than of those prepared 
within the frame-of-reference of technology and com- 
mercialism. Unfortunately, much of the activity in this 
area has been concerned with the development of ma- 
chines rather than with the development of materials for 
use with or without machines. Well over one hundred 
programs are being developed and, as a consequence, the 
respective roles of machines and programs will become 
clearer within a few years. 

Until programmed material is available in reasonable 
amounts, it is difficult to engage in research on the many 
variables which are of great interest to psychologists. Per- 
haps it might be said that one of the most exciting pos- 
sibilities before us is an opportunity for the first time to 
conduct research on the human learning of continuous 
discourse material with reasonable experimental control 
over the instructor variable. Machines are impartial, im- 
personal, and uniform. Teachers, fortunately, are personal 
and individual, but these desirable qualities frequently 
have been confounding variables in educational research 
on teaching methods and the parameters of learning 
under classroom or field conditions. The number of stud- 
ies on variables operating in programmed learning is in- 
creasing rapidly, and soon there will be data to support or 
refute the many ciaims which have been made. 

Our selection of the readings was guided by the several 
considerations set forth in the preceding paragraphs. The 
development and an overview of the field and some of the 
relevant psychological theory are presented in Part I, the 
method—programming—is presented in Part II, and 
samples of research based on the methods of programming 
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are presented in Part III. The nature of teaching ma- 
chines or auto-instructional devices has been minimized 
deliberately in the selection of readings on the grounds 
that at the present stage of development of the field, the 
method for testing the most promising psychology theory 
for a science of teaching is the program which may or may 
not require a machine for presentation of the program 
content. Therefore, descriptions and functions of ma- 
chines will be found only as integral parts of articles, the 
primary concern of which is programming as a method or 
research on the parameters of learning from programmed 
material. 

Each new development in science tends to generate its 
own vocabulary, and this is particularly true of an inter- 
disciplinary subject like programmed learning. To help 
those readers who are unfamiliar with the particular 
terminolgy which has developed for this area, a glossary 
of teaching machine terms is given in Appendix A. 

For those readers who are interested in specific infor- 
mation on some of the existing machines and programs, 
Appendix B provides the names and addresses of several 
machine manufacturers and program distributors. Addi- 
tional information on most of the topics discussed in 
this book can be found in other publications, a list of 
which is also provided in Appendix B. 
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EDITOR’S NOTE 


The set of articles reprinted in this section 
was selected to define the problem and.to 
present some of the issues and the back- 
ground for them. The educational problem 
which has given rise to the great interest in 
teaching machines and programming is ably 
presented by James Finn who has long been 


-a leader in audio-visual education. The arti- 


cles by B. F. Skinner and James Holland de- 
scribe the theory and research in psychology 
upon which much of the recent work on 
programmed learning has been based. A 
somewhat different theoretical framework 
for programmed learning will be found in 
David Zeaman’s article. 

Professor Fry’s article attempts to define 
an issue which perhaps has had more atten- 
tion than it warrants, viz., whether the mode 
of response is a critical variable in learning 
via programmed material. 

A thorough, thoughtful and valuable re- 
view of the work on teaching machines and 
programming from a psychologist’s view is 
provided for the purpose of bridging theory, 
method, and research. 


Part I 


A New Theory for Instructional 
Technology* 


James D. Finn 
University of Southern California 


Technology? relates to education in at least three major 
ways. First, in a society in which science and technology 
are primary, such as America, the society requires that 
the educational system insure an adequate supply of 
scientists and associated technicians. This requirement 
sets a curriculum problem, an organization problem, and 
many other problems associated with the screening, selec- 
tion, and education of young people as potential addi- 
tions to the nation’s technical manpower. 

Second, as a society becomes more and more techno- 
logically oriented and controlled, the question of the 
gencral education of all citizens is raised. The survival 
and management of the whole society theoretically re- 
quire more general education in the sciences and tech- 


* This article was reprinted from Audio Visual Communica- 
tion Review, 1960, 8, 84-94. It is used here with the permission 
of the author. a 

1 For purposes of this paper, no distinction is made between 
“science” and “technology” or between “pure” and “applied” 
science. Thee distinction between “basic” and “applied” re- 
search is perhaps more meaningful, but these two activities are 
still regarded here as positions on a continuum. From a societal 
point of view, it matters little whether the society is hit with 
new ideas or new processes that stem from the ideas, or, as is 
more likely, a combination of the two. 
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curriculum problems, organization problems, and a host 
of nagging, persistent general education problems arise. 

Third, because of the tendency for technology to have 
no limits and constantly r 
inevitable that, in an ad 
nology should begin to 
process itself. As will be 
when education has bee 


nces and, consequently, has 
uum. That vacuum is now 


elationship—the application of 
technology to the educational, or to be more precise, to 


However, the third telationshi 
to merit thorough analysis. 


Again for purposes of this paper, a finer distinction will 
now be drawn. Within the educational Process itself 
there are three general areas in which technology can ‘are 
is being applied. These are: (1) general administration 
(2) testing, and (3) instruction, The uses of technical 
management systems, modern equipment, etc., repre- 
sent the fairly obvious applications to the field of general 
administration. While it probably can easily be shown 
that this area of management is, taken as a whole, two 
or three decades behind its counterpart in industry, the 
problems associated with technology and school man- 
agement are not as difficult as some of the others and 
will not, at this time, be considered. This is not to sa 
that administration is not highly interrelated with the 
other two areas of testing and instruction, but the posi- 
tion can be taken that the problems arising from the 
latter two should guide the technical solutions in adminis- 
tration, not the other way around. 


The second category, testing, is in many respects the 
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most highly developed technology at present existing in 
American education. This is true both from a machine 
and from a systems standpoint. Further, the close rela- 
tionships between psychological, achievement, and other 
types of testing and the instructional process are so well 
defined as to need no comment. However, with some 
exceptions, the technology of the instructional process 
can, for purposes of analysis, be isolated from testing. 
This arbitrary decision is made here in order to further 
the remainder of the discussion. We are left, then, with 
the instructional process by itself and can now turn to 
the impact of technology upon that process. 


DEVELOPMENT OF INSTRUCTIONAL TECHNOLOGY 


The development of a technology of the instructional 
process is relatively new. In the pre-industrial phases of 
both education and industry, while industry was prin- 
cipally at the handwork, artisan level, the instructional 
process relied upon such devices as the slate, the horn- 
book, the blackboard, chalk, and limited single text- 
books with few illustrations. Although attention will 
be confined principally to some of the symbols of tech- 
nology—equipment and machines—it should be em- 
phasized that other factors were in the same state. In 
the area of organization, for example, it is well known 
that the graded school is a late development. 

We note the famous changes in industry at the be- 
ginning of the 19th century—the invention of a group 
of related machines for power, spinning, weaving, etc.— 
which made possible the factory system. Based on the 
work of Toynbee the elder, the term “Industrial Revolu- 
tion” is applied to this period? In education, however, 
the same revolution did not occur and instructional 
technology (with some exceptions here and there) re- 
mained at the pre-industrial level. During the last quar- 
ter-of the century, there was some indication of incipient 
change. Oliver (1956) notes that an exhibit from an 
American school “with maps, charts, textbooks and 
other equipment” wen admiration at the International 
Exposition in Vienna in 1873. He adds that the American 

2 Now called the First Industrial Revolution. 
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school display at Paris in 1878 was even more outstand- 
ing. Significantly, Oliver’s last mention of international 
attention to American instructional technology was at 
Melbourne in 1880. 

By 1900 industry had, in contrast, established factories 
and was moving into assembly line operations; had be- 
gun to apply, in a crude way, research and development 
concepts; had introduced the beginnings of modern man- 
agement; and had developed a sophisticated financial 
system. Developments had reached such a state that the 
perceptive Henry Adams, true prophet and seer of tech- 
nology, foretold the coming of the Age of the Atom and 
the problems it would bring. In the years to 1950, aided 
by the acceleration of two wars, technology burgeoned 
and developed, piled machine upon machine, system upon 
system, added fantastically to its power resources, and 
invented the method of invention. Technology trans- 
formed American society, philosophy, and art. 

During the period from 1900 to 1950, however, tech- 
nology only washed lightly upon the shores of instruc- 
tion. In this same time span, when high speed printing 
techniques, radio, sound motion pictures, television, and 
other pieces of communication technology were invented, 
developed, and exploited, American education failed to 
apply these devices in quantity to the instructional proc- 
ess and, of course, failed to develop the appropriate 
technological systems necessary for this application. There 
were always rumblings, to be sure, as evidenced by the 
statement attributed to Edison in 1916 that the moticn 
picture would replace the teacher. However, looked at 
from the vantage point of 1960, laboratories, 
methods, libraries, and minute arrangements for 
visual materials—the provisions to 1950 Constituted 
what was still a pre-industrial technology for instructi € 

By 1950, industrv had entered what may be called ae 
pre-automation period, which heralded the be iini S 
of a movement toward true automation. Muc ginning 
basic work on computers and other contro} m 
had been done. At this time also, mechanical 
of transporting work goods betwees, machines Systems 
grated some production lines into a 
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dawning, and I have arbitrarily set 1955 as the date at 
which this electronic-mechanical-systems analysis proce- 
dure began to be significantly applied. It is no accident 
that man began to push toward space at about the same - 


` time. 


By 1950, American education had the potentiality (to 
carry on with the analogy) of a mass production tech- 
nology. The hardware—projectors, recorders, television 
—and the materials were present. The systems concepts 
—saturation with audiovisual materials at the point of 
an instructional problem, for example, a concept derived 
from the military experience of World War I]—were 
developed and known in a few audiovisual circles. A 
certain amount of incentive and public acceptance, also 
derived from World War II experience, could be drawn 
upon. The cake of custom, however, proved to be too 
tough. The mass production stage, at least 100 years 
behind industry, was not entered except here and there 
on isolated little islands. 

It was at this point, approximately in 1955, that the 
god came out of the machine in the form of the Ford 
Foundation. Dr. Alvin C. Eurich (and a little later, his 
associate, Dr. Alexander J. Stoddard) gave the instruc- 
tional processes of American education a sharp push into 
mass production technology. The time was ripe. There 
was a shortage of teachers; education and educationists 
were under fire from all sides; Neo-Technocracy was turn- 
ing its attention to education; the race with Russia was 
underway; the natives were restless indeed. 

There are several interesting facets to this shove into 
mass instructional technology. First, television, both 
closed-circuit and broadcast, was chosen as the prime hard- 
ware. This was due, I think, to the unconsciously as- 
sumed basic concept of mass production, usually stated 
in terms of a shortage of teachers, large classes, and 
quality Instruction. Second, the Ford impetus made pro- 
vision, for the first time, for a technology of systems to 
go along with the hardware. Stoddard’s own work in 
designing the school of tomorrow (1957) and the work 
of Trump and his associates in drawing the instructional 
images of the future for the American secondary school 
(1959) are, essentially, proposed systems. 
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Third, the Ford group made use of technology in 
creating the impetus in the first place. The technology 
of social psychology and public relations was drawn upon; 
educational decision-makers were the prime targets, and 
the means used, including high-priced public relations 
counsel, were the best available. This accounts for the 
fact that the teaching profession in general and audio- 
visual specialists in particular—the latter are really the 
only technologists of the profession—were for the most 
part left out of this move into technology. 

Instead, the Foundation went to administrators and 
board members and influential opinion makers of all 
types. Organizations were approached or created: the 
National Educational Television and Radio Center (now, 
I believe, under the administration of the Learning Re- 
sources Institute), the revitalized National Association of 
Educational Broadcasters, the Educational Facilities Lab- 
oratories, Inc., and others. As a sign of this policy to 
short-circuit the profession, it is significant that, except 
for a small grant to promote a seminar or two, the 
Department of Audiovisual Instruction of the National 
Education Association was left out of the largesse, 

The Ford people assumed, I think, that if the cake of 
custom was to be broken the priesthood was not the 
agency to break it. This may mean, in part and as far 
as audiovisual and curriculum people are concerned, that 
to an outsider their technological orientation is at the 
level of the water wheel and the handloom, not the 


computer and the rocket. It is somethin 

On the other hand, it may also mean air a 
Ford Foundation is merely another instrument of E 
Technocracy and that these actions forecast eve eo- 
loss of control of education by the existing a more 
logical profession. Pre-techno- 


IMPACT OF PRESENT TRENDS 


With this background of develo; m ; 

x E ent . 
can now come to grips with the imaat ary In, we 
technology. If we consider education asa E present 
verse, what happens when we introduce cie societal uni- 
universe suddenly and on a larg Igy into that 


parently what is happening. e scale? That is ap- 
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Two general effects can be predicted by basing our 
thinking on two analogous concepts. If the concept of 
entropy holds good in this connection, the educational 
system will become more highly organized and less ran- 
dom in nature (negative entropy will increase); if the 
laws of social change first stated by William G. Sumner 
and William F. Ogbum apply, the introduction of a 
system of technology into the culture will cause dis- 
tortions, stresses, and non-predictable effects to increase 
until a crisis is reached and the culture becomes much 
different. Two historical events serve as examples, one 
taken from Mumford (1955) (the introduction of the 
clock to regulate the habits of Monks) and one from 
Muller (the invention of writing in Egypt for the pur- 
pose of keeping accounts) (1959). Both changed the 
world several times over. The effects of neither were, of 
course, predictable. 

What, then, are the present trends? If you don’t count 
the continuous, grinding, sweating struggle carried on by 
audiovisual specialists, dealers, manufacturers, and pro- 
ducers to introduce, one item at a time, film libraries, 
projectors, recorders, etc., into the educational system in 
the slow process of conversion from a pre-industrial 
technology, two major trends can be identified. These 
two trends, at the moment, lead in opposite directions. 

The first is the trend toward a mass instructional 
technology and is governed by machines and systems 
suitable for that purpose. Foremost, of course, is tele- 
vision, of which there are four instructional types: (1) 
broadcast on an educational channel, (2) broadcast 
on a commercial channel, (3) closed-circuit of the 
Hagerstown-Penn State type in which live instructors 
are used either to supplement instruction or to provide 
direct instruction exclusive of classroom teachers, and 
(4) the Compton type in which filmed lectures are dis- 
tributed via the closed-circuit medium’as replacement for 
classroom teachers. In all cases, the desire is to reach 
more students with fewer teachers or to obtain “quality” 
instruction. 

Mass instruction technology in another form includes, 
of course, the massed film systems. The prime example 
is the EBF series in physics and chemistry, amounting 
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to over 300 half-hour motion pictures intended to be 
used where there are no science teachers, where there are 
teachers not considered qualified, as audiovisual aids to 
more qualified teachers, and with very large groups of 
students. These films, of course, may theoretically be 
used over television. The exploitation of the overhead 
transparency projector is another example of mass instruc- 
tional technology as shown in the Newton, Massachusetts 
experiment in English and grammar and composition. 

In opposition to this trend of mass instruction is a 
growing technology for individual instruction. This trend 
is the audiovisual wave of the future (for the moment). 
The most dramatic development here is that class of 
systems and instruments known as teaching machines. 
Actually, I would class all teaching equipment designed 
for individual or near-individual operation as being in the 
category of teaching machines.3 At present, then, there 
are approximately five types, listed here on an ascending 
scale of sophistication: (1) individual reading papers 
and similar devices, (2) individual viewing and listening 
equipment for existing slides, filmstrips, motion pictures, 
and tecordings, (3) language laboratories of all types, 
(4) specifically programmed printed materials such as 
scrambled textbooks and the Lumsdaine notebook, and 
(5) true teaching machines of the Skinner or Pressey type 
containing carefully worked out verbal or pictorial pro- 
grams with various ingenious mechanical or electronic 


arrangements to test student reaction and inform him 
of his progress, errors, etc. 


EXTRAPOLATION I 


These two major trends toward technologies of mass 
and of individual instruction are not science fiction; the: 
are with us now. Assessment of this impact is ote 
pressing and difficdlt problem. It does not take much of 
a crystal ball to see that a combination of these os 
technologies is the next immediate step. For ex 40 
let us take Maurice Mitchell’s new series for his eu e, 
course on 150 plus films. Let us further assume ete 
3 Preferred terms in some 


« g 4 quarters are “self; T 
“auto-instructional” devices, e “self-instructional” and 
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Mitchell makes a deal with the Rheem Califone Corpo- 
ration to program their teaching machine with material 
to go along with the chemistry films. (Mr. Eurich might 
have to put up the money for the initial effort.) Let us 
continue by saying we will project the films hourly on the 
Compton College closed-circut television film distribu- 
tion system and place the machines in proper cubby- 
holes for Compton College students to work at specified 
times. We could get, almost immediately, to what can 
be called total educational automation. 

Or, in another context, let us suppose that the new 
Ford creation, the Learning Resources Institute, initiates 
another Continental Classroom in mathematics over na- 
tional broadcast television; and that the new audiovisual 
teaching machines now being developed are programmed 
to relate to this new series; and that these machines are 
placed by the foundation in every public library in 
America; and that, at the conclusion of the instructional 
period, students report to designated centers where the 
Educational Testing Service examines them and certifies 
them to the colleges of their choice. Think, for a mo- 
ment, about that one. It is now possible not only to 
eliminate the teachers but the school system.4 

These may be considered extreme applications, but 
such applications, at least experimentally, are inevitable. 
In fact, one such study, testing the combination of these 
two technologies, is now underway. The point is that 
both the mass instruction systems and the technology 
of individual instruction—teaching machines—are getting 
terrific momentum. These technologies are going to hit 
education with a million-pound thrust. What will be the 
effect upon our educational society? What is the role of 
the teacher, the audiovisual specialist, the curriculum 
director? Quo vadis the curriculum itself? This last ques- 
tion is particularly pertinent if the medium or image 
governs the message, as McLuhan maintains. 


4 Time magazine had some fun lifting this sentence out of 
context. See Time, March 14, 1959, p. 68. Time also said: 
“Finn offered one cavezt: this awesome equipment must not 
fall into the hands of any one private institution, e.g., the Ford 
Foundation. Said he, “The American people don’t elect repre- 
sentatives to the Ford Foundation. 
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Technology never sits still. The extrapolations A 
in the last section are immediate. What of tor wil 
can be made for a time slightly farther into the fu Fé 
see several developments. First, it is reasonable to ig 
that programming for teaching machines will mE anche 
the verbal-Socratic-Skinner type to the audiovisual-br: adl 
ing type. That is, the machines will present canta 
content based upon student pretests, using films, s The 
filmstrips, tapes, and/or videotape as the medium. aa 
Presentation sequences will be longer and the pee 
will be given an Opportunity to select additional rag a 
for further explanation if the machine, through tes rhe 
informs him that he needs it. Records, of course, wil 
maintained instantaneously by miniaturized compusi 

Secondly, the work in England, of Ashby, Pask, an : 
others with teaching machines based on the design oor 
cepts of biological computers will affect the technology ©} 
the present teaching machines much a 
chology affects stimulus-response psychology. Pask has 


already prod , In training key punch 


maintains he can develop 
such a machine to teach decision-making with present 
hardware and know-how (1958), 

A second development 

with the Airborne TV experime 
tion over a city in Indi 
more airplanes carrying 
can broadcast signals over much 
conventional tower transmission, € programs will go 
into several states and, because of the location, into both 
urban and rural areas and into innumerable heretofore 
independent school districts, Effects of this on local 
control of curricula, curriculum Planning Procedures, 
images of master teachers, patterns of School organiza. 
tion, etc., have yet to be speculated abo 


z ut, let alone ex- 
plored. Nothing technical would prevent, by the way, 


€teater distance than 


e 
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a national network of such aircraft as an addition to our 
other national meaus of transmission. 

A third predicted development is based on information 
that it is technically possible, although not yet econom- 
ically feasible, to produce a motion picture film with 
characteristics similar to the Polaroid film that is instan- 
taneously developable. What this would do to videotape, 
to production techniques, and to the possibilities of 
local production of materials is of great interest to the 
audiovisual field.’ 


TEACHER CONTROL 


The fourth development that can be anticipated is 
the most spectacular. Essentially, it is based on a sys- 
tems concept. It is theoretically possible now to design 
an automatic classroom under the control of the teacher. 
Most of the elements are present or can be designed. 
Such a classroom would have total light and air control, 
automatic projection and television systems, technical 

rovision for the best possible dicussion environment, dis- 
play situations, etc., which could be changed at will. By 
planned programming, the classroom could be made to 
function for major presentations, small group discus- 
sions, individual work at teaching machines, creative 
periods, and the like. As we said, all of this could be 
under the control of the teacher. The classroom then 
would become the teaching machine. Adrian TerLouw at 
Eastman Kodak has done some work in this area and it 
is, of course, implied in the provocative article by Simon 
Ramo which is so often cited in discussions of teaching 
machines (1957). An acquaintance of mine refers to 
this concept as the “mad scientist” classroom. 


IMPLICATIONS ` 


To assess the implications of this current and predicted 
impact of technology on the instructional process is not 
easy. As with any such speculation, it is also hazardous. 
It is, however, a job we must attempt. 


6 ew thermo-plastic method of image-recording devel- 
gaa General Electric can do much the same thing. 
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We have to start, I think, with the proposition that 
our educational society is in the position of a backward 
or underdeveloped culture suddenly assailed by the 20th- 
century engineer. In addition, reactions in some profes- 
sional circles to the advent of television are similar to 
those factory workers of the 19th century who attempted 
to destroy the machines that were taking over their jobs. 
In the long run, the machines remained; instructional 
technology is, no doubt, here to stay. Our problem be- 
o live with it on some 
how to control it so 
cation may be served 
ntral in the process. 
think the concept of 
he thrust and energy of 
organization upon us at 
lied to instruction. Such 


that the proper objectives of edu 
and the human being remain ce 

Certain things are obvious. T 
negative entropy® will hold. TH 
technology will force a greater 
every point at which it is app 


states and in those many schoo 
The closed-circuit experiment b 


tigor i 
never been felt before in Anaheim. oich ha 
Programming, in its machin 


» teaching mach; 
language laboratories. This me. 8 Machine 


needed. Who is to do this? 5 x 
old sense, both a curriculum a maa 
‘ 
*In all the tests I have been able o . 
holds. I am almost convinced that this sink of, aus concept 
status as, forgive me, “Finn's law.” achieve some 
7 The role of energy in relation to AV 
needs much study, For ee th 
negative entropy concept s ould be ex. mined in 
same way that Cottrell studied energy in much the 
and Society. general in Energy 
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It is also a social problem. The heartland is programming. 
He who controls the programming heartland controls the 
educational system. Will it be a foundation, a committee 
Or scientists, textbook publishers and film producers, the 
NEA, the school superintendent, the board of education, 
the students, or the general public? . 

Too, how many of us will go overboard and sink with 
the old concepts that will be absorbed or outmoded and 
tossed to the sharks by the new technology? Take the 
concept of instructional materials which some curriculum 
specialists love so well and which caused quite a con- 
troversy in DAVI circles a few years ago. The fight was 
futile, as is the love. The concept of programming and 
the systems and systems analysis it implies completely 
absorbs the idea of materials. Instructional materials 
becomes an outmoded, atomistic, pre-technological con- 
cept useful mainly to the historians of education.® 

The concept of audiovisual education may accompany 
instructional materials down the drain, or it may not, 
depending on whether or not it can be redefined ac- 
ceptably.9 The Skinner and Pressey type teaching ma- 
chines and their descendants are, for example, primarily 
verbal devices, and yet their management and program- 
ming as technical electronic devices belong somewhere. 
At a practical level, within a school situation, someone 
is going to have to worry about them. It is my conten- 
tion that the audiovisual field is in the easiest position 
to help integrate these mechanisms properly into the 


instructional process. They are not primarily audio- 
visual; they are primarily technological. The AV field, I 
$ w up. We, the audiovisual 


think, must now suddenly gro’ eter 
specialists, are, of all educational personnel, the closes 
to technology now; we have, I think, to become specialists 


in learning technology—and that’s how I would redefine 


iovi ion. A Se 
E Medanr crcl people? “Their work is just 
beginning. It can become infinitely more rigorous and 


8 McLuhan à maintains the same thing about audiovisual 
aids (1959). nd terminology) of the audiovisual field 
» Gontemplated G subject of a future special supplement to 


em] 
AV oe rUNICATION REVIEW. 
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at the same time more satisfying. I suspect some of 
the emotional Rousseauians will have to get out of the 
curriculum business. The rest can step in and face this 
problem of programming a learning technology in human 
terms and find great excitement and reward. The profes- 
sion as a whole must be made to sense this powerful 
movement to instructional technology and be made 


teady to seize the great Opportunity it offers to make 
all teachers highly professional, 


THE PHILOSOPHY OF ADVENTURE 


In closing, I should like to quote Whitehead, who 
said that it is the business of the f 


when men were free to roam instead of oe anye 
damn hole in the ground” (p. 6 ). The “sood Hi ee 
are gone; approached with intellige 2 

of the future will be better, ence and zest, the days 
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The Science of Learning and the 
Art of Teaching* 


B. F. SKINNER 
Harvard University 


Some promising advances have recent! been made in 
the field of learning. Special techniques have been de- 
signed to arrange what are called “contingencies of 
reinforcement” —the relations which prevail between be- 


maze, for example, or in the T-maze, t 
or the familiar discrimination apparatus. 
duced by the organism’s behavior are ] 
tuating circumstances. There js man 
the turn-to-the-right and the food-cup 
alley. It is not surprising that techni ue nes 

have yielded only very rough data from which ae sort 
formities demanded by an experimen e uni- 


* Paper presented at a conference o 3 À 
Psychology and the Behavioral Sciences rat d he Ung tends in 
Pittsburgh, March 12, 1954. This article i repi eo ity OF 
the Haryard Educational Review, 1954, 9. punted from 
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extracted only by averaging many cases. In none of this 
work has the behavior of the individual organism been 
predicted in more than a statistical sense. The learning 
processes which are the presumed object of such research 
are reached only through a series of inferences. Current 
preoccupation with deductive systems reflects this state 
of the science. 

Recent improvements in the conditions which control 
behavior in the field -of learning are of two principal 
sorts. The Law of Effect has been taken seriously; we 
have made sure that effects do occur and that they occur 
under conditions which are optimal for producing the 
changes called learning. Once we have arranged the 
particular type of consequence called a reinforcement, . 
our techniques permit us to shape up the behavior of an 
organism almost at will. It has become a routine exercise 
to demonstrate this in classes in elementary psychology 
by conditioning such an organism as a pigeon. Simply by 
presenting food to a hungry pigeon at the right time, it 
is possible to shape up three or four well-defined responses 
in a single demonstration period—such responses as 
turning around, pacing the floor in the pattern of a figure- 
8, standing still in a corner of the demonstration appara- 
tus, stretching the neck, or stamping the foot. Extremely 
complex performances may be reached through succes- 
sive stages in the shaping process, the contingencies of 
teinforcement being changed progressively in the direc- 
tion of the required behavior. The results are often quite 
dramatic, In such a demonstration one can see learning 
take place. A significant change in behavior is often ob- 
vious as the result of a single reinforcement. pi 

A second important advance in technique permits us 
to maintain behavior in given states of strength for long 
periods of time. Reinforcements continue to be impor- 
tant, of course, long after an organism has learned how 
to do something, long after it has- acquired behavior. 
They are necessary to maintain the behavior in strength. 
Of special interest 1s the effect of various schedules of 
intermittent reinforcement. Charles B. Ferster and the 
author are currently. preparing an extensive report of a 
five-year research program, sponsored by the Office of 
Naval Research, in which most of the important types of 
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assigned to the field of motivation, although the prin- 
cipal operation is simply the arrangement of contin- 


& up to this 
ne study of the 
behavior, It is 
Series of contin- 


stable state are excellent material for t} 
factors important in nonsynthetic social 


1 The reader may wish to review Dr. Skinner's article, “g, 
Contributions of an Experimenta] Analysis of Behavior to 
Psychology as.a Whole,” The American P. Sychologist, 1953, 
8, 69-78. Ed. 
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gencies could afte Si Me -ed dhe human organism 


through the evolution of cultural patterns. 

-Cooperation can also be set up, perhaps more easily 
than competition. We have trained two pigeons to co- 
ordinate their behavior in a cooperative endeavor with a 
precision which equals that of the most skillful human 
dancers. In a more serious vein these techniques have 
permitted us to explore the complexities of the individual 
organism and to analyze some of the serial or coordinate 
behaviors involved in attention, problem solving, various 
types of self-control, and the subsidiary systems of te- 
sponses within a single organism called “personalities.” 
Some of these are exemplified in what we call multiple 
schedules of reinforcement. In general a given schedule 
has an effect upon the rate at which a response is emitted. 
Changes in the rate from moment to moment show 
a pattern typical of the schedule. The pattern may be as 
simple as a constant rate of responding at a given value, 
it may be a gradually accelerating rate between certain 
extremes, it may be an abrupt change from not respond- 
ing at all to a given stable high rate, and so on. It has 
been shown that the performance characteristic of a 
given schedule can be brought under the control of a 
particular stimulus and that different performances can 
be brought under the control of different stimuli in 
the same organism. At a recent meeting of the American 
Psychological Association, Dr. Ferster and the author 
demonstrated a pigeon whose behavior showed the pattern 
typical of “fixed-interval” reinforcement in the presence 
of one stimulus and, alternately, the pattern typical of 
the very different schedule called “fixed ratio” in the 
presence of a second stimulus. In the laboratory we 
have been able to obtain performances appropriate to 
nine different schedules in the presence of appropriate 
stimuli in random alternation. When, Stimulus 1 is pres- 
ent, the pigeon executes the performance appropriate to 
Schedule 1. When Stimulus 2 is present, the pigeon 
executes the:performance appropriate to Schedule 2. And 
so on. This result is important because itsmakes the extra- 

olation of our laboratory results to daily: life much more 
plausible. We are all constantly shifting from schedule 


to schedule as our te environment changes, but 
yeh J, VAS 2 Wes B7E 3 
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the dynamics of the control exercised by reinforcement 
remain essentially unchanged. 

It is also possible to construct very complex sequences 
of schedules. It is not easy to describe these ina few 
words, but two or three examples may be mentioned. In 
one experiment the pigeon generates a performance 
appropriate to Schedule A where the reinforcement is 
simply the production of the stimulus characteristic of 
Schedule B, to which the pigeon then responds appro- 
priately. Under a third stimulus, the bird yields a perform- 
ance appropriate to Schedule C where the reinforcement 
in this case is simply the production of the stimulus 
characteristic of Schedule D, to which the bird then 
responds appropriately. In a special case, first investigated 
by L. B. Wyckoff, Jr., the organism responds to one 
stimulus where the reinforcement consists of the clarifi- 
cation of the stimulus controlling another response. The 
first response becomes, so to speak, an objective form 
of “paying attention” to the second stimulus. In one 
important version of this experiment, as yet unpublished, 
we could say that the pigeon is telling us whether it is 
“paying attention” to the shape of a spot of light or to its 
color. 

One of the most dramatic applications of these tech- 
niques has recently been made in the Harvard Psycho- 
logical Laboratories by Floyd Ratliff and Donald S. 
Blough, who have skillfully used multiple and serial 
schedules of reinforcement to study complex perceptual 
processes in the infrahuman organism. They have achieved 
a sort of psycho-physics without verbal instruction. In a 
tecent experiment by Blough, for example, a pigeon 
draws a detailed dark-adaptation curve showing the char- 
acteristic breaks of rod and cone vision. The curve is 
recorded continuously in a single experimental period and 
is quite comparable with the curves of human subjects. 
The pigeon behaves in a way which, in the human case, 
we would not hesitate to describe by saying that it adjusts 
a very faint patch of light until it can just be seen. 

In all this work, the species of the organism has made 
surprisingly little difference. It is trae that the organisms 
studied have all been vertebrates, but they still cover a 
wide range. Comparable results have been obtained with 
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pigeons, rats, dogs, monkeys, human children, and most 
recently, by the author in collaboration with Ogden R. 
Lindsley, human psychotic subjects. In spite of greai 
phylogenetic differences, all these organisms show amaz: 
ingly similar properties of the learning process. It should 
be emphasized that this has been achieved by analyzing 
the effects of reinforcement and by designing techniques 
which manipulate reinforcement with considerable preci- 
sion. Only in this way can the behavior of the individual 
organism be brought under such precise control. It is also 
important to note that through a gradual advance to 
complex interrelations among responses, the same degree 
of rigor is being extended to behavior which would usually 
be assigned to such fields as perception, thinking, and 
personality dynamics. 

From this exciting prospect of an advancing science of 
learning, it is a great shock to turn to that branch of 
technology which is most directly concerned with the 
learning process—education. Let us consider, for ex- 
ample, the teaching of arithmetic in the lower grades. 
The school is concerned with imparting to the child a 
large number of responses of a special sort. The responses 
are all verbal. They consist of speaking and writing cer- 
tain words, figures, and signs which, to put it roughly, 
refer to numbers and to arithmetic operations. The first 
task is to shape up these responses—to get the child to 
pronounce and to write responses correctly, but the prin- 
cipal task is to bring this behavior under many sorts of 
stimulus control. This is what happens when the child 
learns to count, to recite tables, to count while ticking 
off the items in an assemblage of objects, to respond to 
spoken or written numbers by saying “odd,” “even,” 
“prime,” and so on. Over and above this elaborate reper- 
toire of numerical behavior, most of which is often dis- 
missed as the product of rote learning, the teaching of 
arithmetic looks forward to those complex serial arrange- 
ments of responses involved in original mathematical 
thinking. Thechild must acquire responses of transposing, 
clearing fractions, and so on, which modify the order or 
pattern of the original material so that the response called 
a solution is eventually made possible. 

Now, how is this extremely complicated verbal reper- 
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toire set up? In the first place, what reinforcements are 
used? Fifty years ago the answer would have been clear. 
At that time educational control was still frankly aver- 
sive. The child read numbers, copied numbers, memorized 
tables, and performed operations upon numbers to escape 
the threat of the birch rod or cane. Some positive rein- 
forcements were perhaps eventually derived from the 
increased efficiency of the child in the field of arithmetic 
and in rare cases some automatic reinforcement may have 
tesulted from the sheer manipulation of the medium— 
from the solution of problems or the discovery of the 
intricacies of the number system. But for the immediate 
purposes of education the child acted to avoid or escape 
punishment. It was part of the reform movement known 
as progressive education to make the positive conse- 
quences more immediately effective, but any one who 
visits the lower grades of the average school today will 
observe that a change has been made, not from aversive 
to positive control, but from one form of aversive stimu- 
lation to another. The child at his desk, filling in his 
workbook, is behaving primarily to escape from the threat 
of a series of minor aversive events—the teacher’s dis- 
pleasure, the criticism or ridicule of his classmates, an 
ignominious showing in a competition, low marks, a trip 
to the office “to be talked to” by the principal, or a word 
to the parent who may still resort to the birch rod. In 
this welter of aversive consequences, getting the right 
answer is in itself an insignificant event, any effect of 
which is lost amid the anxieties, the boredom, and the 
aggressions which are the inevitable by-products of aver- 
sive control.2 

Secondly, we have to ask how the contingencies of 
reinforcement are arranged. When is a numerical opera- 
tion reinforced as “right”? Eventually, of course, the 
pupil may be ablesto check his own answers and achieve 
some sort of automatic reinforcement, but in the early 
stages the reinforcement of being right is usually accorded 
by the teacher. The contingencies she provides are far , 
from optimal. It can easily be demonstrated that, unless 
explicit mediating behavior has been set up, the lapse of 


*Sxinner, B. F. Science and Human Behavior. New York: 
Macmillan, 1953. 
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only a few seconds between response and reinforcement 
destroys most of the effect. In a typical classroom, never- 
theless, long periods of time customarily elapse. The 
teacher may walk up and down the aisle, for example, 
while the class is working on a sheet of problems, pausing 
here and there to say right or wrong. Many seconds or 
minutes intervene between the child’s response and the 
teacher’s reinforcement. In many cases—for example, 
when papers are taken home to be corrected—as much as 
24 hours may intervene. It is surprising that this system 
has any effect whatsoever. 

A third notable shortcoming is the lack of a skillful 
program which moves forward through a series of progres- 
sive approximations to the final complex behavior de- 
sired. A long series of contingencies is necessary to bring 
the organism into the possession of mathematical be- 
havior most efficiently. But the teacher is seldom able 
to reinforce at each step in such a series because she 
cannot deal with the pupil’s responses one at a time. It 
is usually necessary to reinforce the behavior in blocks 
of responses—as in correcting a work sheet or page from 
a workbook. The responses within such a block must 
not be interrelated. The answer to one problem must 
not depend upon the answer to another. The number of 
stages through which one may progressively approach a 
complex pattern of behavior is therefore small, and the 
task so much the more difficult. Even the most modern 
woskbook in beginning arithmetic is far from exem- 
plifying an efficient program for shaping up mathematical 
behavior. 

Perhaps the most serious criticism of the current class- 
toom is the relative infrequency of reinforcement. Since 
the pupil is usually dependent upon the teacher for 
being right, and since many pupils are usually dependent. 
upon the same teacher, the total number of contingencies 
which may be arranged during, say, the first four years, 
is of the order of only a few thousand. But a very rough 
estimate suggests that efficient mathematical behavior 
at this level requires something of the order of 25,000 
contingencies. We may suppose that even in the brighter 
student a given contingency must be arranged several 
times to place the behavior well in hand. The responses 
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to be set up are not simply the various items in tables of 
addition, subtraction, multiplication, and division; we 
have also to consider the alternative forms in which 
each item may be stated. To the learning of such ma- 
terial we should add hundreds of responses concerned 
with factoring, identifying primes, memorizing series, 
using short-cut techniques of calculation, constructing 
and using geometric representations or number forms, 
and so on. Over and above all this, the whole mathe- 
matical repertoire must be brought under the control of 
concrete problems of considerable variety. Perhaps 50,000 
contingencies is a more conservative estimate. In this 
frame of reference the daily assignment in arithmetic 
seems pitifully meagre. 

The result of all this is, of course, well known. Even 
our best schools are under criticism for their inefficiency 
in the teaching of drill subjects such as arithmetic. The 
condition in the average school is a matter of wide- 
spread national concern. Modern children simply do not 
learn arithmetic quickly or well. Nor is the result simply 
incompetence. The very subjects in which modern tech- 
niques are weakest are those in which failure is most con- 
spicuous, and in the wake of an ever-growing incom- 
petence come the anxicties, uncertainties, and aggressions 
which in their turn present other problems to the school. 
Most pupils soon claim the asylum of not being “ready” 
for arithmetic at a given level or, eventually, of not hav- 
ing a mathematical mind. Such explanations are readily 
seized upon by defensive teachers and parents. Few 
pupils ever reach the stage at which automatic reinforce- 
ments follow as the natural consequences of mathemati- 
cal behavior. On the contrary, the figures and symbols 
of mathematics have become standard emotional stimuli. 
The glimpse of a column of figures, not to say an alge- 
braic symbol or ax integral sign, is likely to set off—not 
mathematical behavior—but a reaction of anxiety, guilt, 
or fear. 

The teacher is usually no happier about*this than the 
pupil. Denied the opportunity to control via the birch 
tod, quite at sea as to the mode of operation of the few 
techniques at her disposal, she spends as little time as 
possible on drill subjects and eagerly subscribes to phi- 
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losophies of education which emphasize material of 
greater inherent interest. A confession of weakness is her 
extraordinary concern lest the child be taught something 
unnecessary. The repertoire to be imparted is carefully 
reduced to an essential minimum. In the field of spelling, 
for example, a great deal of time and energy has gone 
into discovering just those words which the young child is 
going to use, as if it were a crime to waste one’s educa- 
tional power in teaching an unnecessary word. Even- 
tually, weakness of technique emerges in the disguise of 
a reformulation of the aims of education. Skills are 
minimized in favor of vague achievements—educating 
for democracy, educating the whole child, educating for 
life, and so on. And there the matter ends; for, unfor- 
tunately, these philosophies do not in turn suggest im- 
provements in techniques. They offer little or no help 
in the design of better classroom practices. 

There would be no point in urging these objections if 
improvement were impossible. But the advances which 
have recently been made in our control of the learning 
process suggest a thorough revision of classroom practices 
and, fortunately, they tell us how the revision can be 
brought about. This is not, of course, the first time that 
the results of an experimental science have been brought 
to bear upon the practical problems of education. The 
modern classroom does not, however, offer much evi- 
dence that research in the field of learning has been re- 
spected or used. This condition is no doubt partly due 
to the limitations of earlier research. But it has been 
encouraged by a too hasty conclusion that the laboratory 
study of learning is inherently limited because it cannot 
take into account the realities of the classroom. In the 
light of our increasing knowledge of the learning process 
we should, instead, insist upon dealing with those reali- 
ties and forcing a substantial change in them. Education 
is perhaps the most important branch of scientific tech- 
nology. It deeply affects the lives of all of us. We can no 
longer allow the exigencies of a practical situation to sup- 
press the tremendous improvements which are within 
teach. The practical situation must be changed. 

There are certain questions which have to be answered 
in turning to the study of any new organism. What be- 


28 SCIENCE OF LEARNING AND Art OF TEACHING 


havior is to be set up? What reinforcers are at hand? 
What responses are available in embarking upon a pro- 
gram of progressive approximation which will lead to the 
final form of the behavior? How can reinforcements be 
most efficiently scheduled to maintain the behavior in 
strength? These questions are all relevant in considering 
the problem of the child in the lower grades. 

In the first place, what reinforcements are available? 
What does the school have in its possession which will 
reinforce a child? We may look first to the material to be 
learned, for it is possible that this will provide consider- 
able automatic reinforcement. Children play for hours 
with mechanical toys, paints, scissors and paper, noise- 
makers, puzzles—in short, with almost anything which 
feeds back significant changes in the environment and is 
teasonably free of aversive properties. The sheer control 
of nature is itself reinforcing. This effect is not evident 
in the modem school because it is masked by the emo- 
tional responses generated by aversive control. It is true 
that automatic reinforcement from the manipulation of 
the environment is probably only a mild reinforcer and 
may need to be carefully husbanded, but one of the most 
striking principles to emerge from recent research is that 
the net amount of reinforcement is of little significance. 
A very slight reinforcement may be tremendously effec- 
tive in controlling behavior if it is wisely used. 

If the natural reinforcement inherent in the subject 
matter is not enough, other reinforcers must be em- 
ployed. Even in school the child is occasionally permitted 
to do “what he wants to do,” and access to reinforce- 
ments of many sorts may be made contingent upon the 
more immediate consequences of the behavior to be 
established. Those who advocate competition as a useful 
social motive may wish to use the reinforcements which 
follow from excelling others, although there is the diffi- 
culty that in this case the reinforcement of one child is 
necessarily aversive to another. Next in order we might 
place the good will and affection of the teacher, and 


only when that has failed need we turn to the use of 
aversive stimulation. é 


In the second 


nd place, how are these reinforcements to 
be made continge 


nt upon the desired behavior? There are 
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two considerations here—the gradual elaboration of ex- 
tremely complex patterns of behavior and the mainte- 
nance of the behavior in strength at each stage. The 
whole process of becoming competent in any field must 
be divided into a very large number of very small steps, 
and reinforcement must be contingent upon the accom- 
plishment of each step. This solution to the problem of 
creating a complex repertoire of behavior also solves the 
problem of maintaining the behavior in strength. We 
could, of course, resort to the techniques of scheduling 
already developed in the study of other organisms but in 
the present state of our knowledge of educational prac- 
tices, scheduling appears to be most effectively arranged 
through the design of the material to be learned. By 
making each successive step as small as possible, the fre- 
quency of reinforcement can be raised to a maximum, 
while the possibly aversive consequences of being wrong 
are reduced to a minimum. Other ways of designing 
material would yield other programs of reinforcement. 
Any supplementary reinforcement would probably have 
to be scheduled in the more traditional way. 

These requirements are not excessive, but they are 
probably incompatible with the current realities of the 
classroom. In the experimental study of learning it has 
been found that the contingencies of reinforcement 
which are most efficient in controlling the organism 
cannot be arranged through the personal mediation of 
the experimenter. An organism is affected by subtle 
details of contingencies which are beyond the capacity of 
the human organism to arrange. Mechanical and electri- 
cal devices must be used. Mechanical help is also de- 
manded by the sheer number of contingencies which 
may be used efficiently in a single experimental session. 
We have recorded many millions of responses from a 
single organism during thousands of experimental hours. 
Personal arrangement of the contingències and personal 
observation of the results are quite unthinkable. Now, 
the human organism is, if anything, more sensitive to 
precise contingencies than the other organisms we have 
studied. We have every reason to expect, therefore, that 
the most effective control of human learning will require 
instrumental aid. The simple fact is that, as a mere re- 
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inforcing mechanism, the teacher is out of date. This 
would be true even if a single teacher devoted all her 
time to a single child, but her inadequacy is multiplied 
many-fold when she must serve as a reinforcing device to 
many children at once. If the teacher is to take advantage 
of recent advances in the study of learning, she must have 
the help of mechanical devices. . 

The technical problem of providing the necessary in- 
strumental aid is not particularly difficult. There are 
many ways in which the necessary contingencies may be 
arranged, either mechanically or electrically. An inexpen- 
sive device which solves most of the principal problems 
has already been constructed. It is still in the experimen- 
tal, stage, but a description will suggest the kind of in- 
strument which seems to be required. The device con- 
sists of a small box about the size of a small record plaver. 
On the top surface is a window through which a question 
or problem printed on a paper tape may be seen. The 
child answers the question by moving one or more sliders 
upon which the digits 0 through 9 are printed. The an- 
swer appears in square holes punched in the paper upon 
which the question is printed. When the answer has been 
set, the child turns a knob. The operation is as simple as 
adjusting a television set. If the answer is right, the knob 
turns freely and can be made to ting a bell or provide 
Some other conditioned reinforcement. If the answer is 
wrong, the knob will not turn. A counter may be added 
to tally wrong answers. The knob must then be reversed 
slightly and a second attempt at a right answer made. 
(Unlike the flash-card, the device reports a wrong an- 
swer without giving the tight answer.) When the answer 
is right, a further turn of the knob engages a clutch 
which moves the next problem into place in the window. 
This movement cannot be completed, however, until the 
sliders have been returned to zero. 

The important features of the device are these: Re- 
inforcement for the right answer is immediate. The mere 
manipulation of the device will probably te reinforcing 
enough to keep the average pupil at work for a suitable 
period each day, provided traces of earlier aversive con- 
trol can be wiped out. A teacher may supervise an entire 
class at work on such devices at the same time, yet each 
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child may progress at his own rate, completing as many 
problems as possible within the class period. If forced to 
be away from school, he may return to pick up where 
he left off. The gifted child will advance rapidly, but can 
be kept from getting too far ahead either by being ex- 
cused from arithmetic for a time or by being given special 
sets of problems which take him into some of the inter- 
esting bypaths of mathematics. 

The device makes it possible to present carefully de- 
signed material in-which one problem can depend upon 
the answer to the preceding and where, therefore, the 
most efficient progress to an eventually complex reper- 
toire can be made. Provision has been made for recording 
the commonest mistakes so that the tapes can be modi- 
fied as experience dictates. Additional steps can be in- 
serted where pupils tend to have trouble, and ultimately 
the material will reach a point at which the answers of 
the average child will almost always be right. y 

If the material itself proves not to be sufficiently rein- 
forcing, other reinforcers in the possession of the teacher 
or school may be made contingent upon the operation 
of the device or upon progress through a series of prob- 
lems. Supplemental reinforcement would not sacrifice the 
advantages gained from immediate reinforcement and 
from the possibility of constructing an optimal series of 
steps which approach the complex repertoire of mathe- 
matical behavior most efficiently. 

A similar device in which the sliders carry the letters of 
the alphabet has been designed to teach spelling. In addi- 
tion to the advantages which can be gained from precise 
reinforcement and careful programming, the device will 
teach reading at the same time. It can also be used to 
establish the large and important repertoire of verbal 
relationships encountered in logic and science. In short, 
it can teach verbal thinking. As to content instruction, 
the device can be operated as a multiple-choice self-rater. 

Some objections to the use of such devices in the 
classroom cea easily be foreseen. The cry will be raised 
that the child is being treated as a mere animal and that 
an essentially human intellectual achievement is being 
analyzed in unduly mechanistic terms. Mathematical 
behavior is usually regarded, not as a repertoire of re- 
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sponses involving numbers and numerical operations, but 
as evidences of mathematical ability or the exercise of the 
power of reason. It is true that the techniques which 
are emerging from the experimental study of learning are 
not designed to “develop the mind” or to further some 
vague “understanding” of mathematical relationships. 
They are designed, on the contrary, to establish the very 
behaviors which are taken to be the evidences of such 
mental states or processes. This is only a special case of 
the general change which is under way in the interpreta- 
tion of human affairs. An advancing science continues to 
offer more and more convincing alternatives to traditional 
formulations. The behavior in terms of which human 
thinking must eventually be defined is worth treating in 
its own right as the substantial goal of education. 

Of course the teacher has a more important function 
than to say right or wrong. The changes proposed would 
free her for the effective exercise of that function. Mark- 
ing a set of papers in arithmetic—“Yes, nine and six are 
fifteen; no, nine and seven are not eighteen”—is beneath 
the dignity of any intelligent individual. There is more 
important work to be done—in which the teacher’s rela- 
tions to the pupil cannot be duplicated by a mechanical 
device. Instrumental help would merely improve these 
relations. One might say that the main trouble with 
education in the lower grades today is that the child is 
obviously not competent and knows it and that the 
teacher is unable to do anything about it and knows that 
too. If the advances which have tecently been made in 
our control of behavior can give the child a genuine com- 
petence in reading, writing, spelling, and arithmetic, 
then the teacher may begin to function, not in lieu of a 
cheap machine, but through intellectual, cultural, and 
emotional contacts of that distinctive sort which testify 
to her status as a human being. 

J Another Possible objection is th 
tion will mean technologi 


go around and until the hours and e 
the teacher are comparable to thos 
employment. Mechanical 
tiresome labors of the te 
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sarily shorten the time during which she remains in 
contact with the pupil. 

A more practical objection: Can we afford to mecha- 
nize our schools? The answer is clearly yes. The device I 
have just described could be produced as cheaply as a 
small radio or phonograph. There would need to be far 
fewer devices than pupils, for they could be used in rota- 
tion. But even if we suppose that the instrument eventu- 
ally found to be most effective would cost several hundred 
dollars and that large numbers of them would be re- 
quired, our economy should be able to stand the strain. 
Once we have accepted the possibility and the necessity 
of mechanical help in the classroom, the economic prob- 
lem can easily be surmounted. There is no reason why 
the school room should be any less mechanized than, 
for example, the kitchen. A country which annually pro- 
duces millions of refrigerators, dish-washers, automatic 
washing-machines, automatic clothes-driers, and automa- 
tic garbage disposers can certainly afford the equipment 
necessary to educate its citizens to high standards of com- 
petence in the most effective way. 

There is a simple job to be done. The task can be 
stated in concrete terms. The necessary techniques are 
known. The equipment needed can easily be provided. 
Nothing stands in the way but cultural inertia. But what 
is more characteristic of America than an unwillingness 
to accept the traditional as inevitable? We are on the 
threshold of an exciting and revolutionary period, in 
which the scientific study of man will be put to work in 
man’s best interests. Education must play its part. It 
must accept the fact that a sweeping revision of educa- 
tional practices is possible and inevitable. When it has 
done this, we may look forward with confidence to a 
school system which is aware of the nature of its tasks, 
secure in its methods, and generously supported by the 
informed and effective citizens whom education itself 


will create. 
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Teaching Machines: An Application 
of Principles From the Laboratory* 
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Much has been said of teaching machines tecently— 
but the emphasis has tended to be on the gadgets rather 
than on the much more significant development of a new 
technology of education initiated by B. F. Skinner 
(1954). The technology does use a device called a teach- 
ing machine which presents a finely graded series of 
problems and provides immediate “reward” or teinforce- 
ment for the student’s correct answers. But emphasis on 
machines has tended to obscure the more important 
facets of the new technology based on application of 
Principles from the laboratory. The machines of today 
are not necessarily better than those of yesterday. Indeed, 
adequate machines could have been built hundreds of 
years ago. The movement today is not simply the mecha- 
nization of teaching, but instead the development of a 
new technology—a behavioral engineering of teaching 
procedures. 


The history of nnsuccessful teaching machines illus- 
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trates the relatively greater importance of the technique 
as opposed to the gadgets. The first teaching machine 
was patented 93 years ago. There have since been a series 
of patents and a promising burst of activity by Sidney 
Pressey and his associates (1926). None of these early 
efforts really caught hold. But during this period in which 
the idea of mechanized teaching has been latent the 
science of behavior has developed principles which per- 
mit extremely precise control of behavior. This new 
technology is not only the so-called automation of teach- 
ing but is an attempt to obtain the kind of behavioral 
control shown possible in the laboratory. 

We have, of course, seen other practical applications 
of scientific psychology. We are all familiar with the 
development of a technology of testing which permits 
placing an individual in situations suited to his abilities. 
We are also familiar with another technology called hu- 
man engineering, which fits machines and jobs to the 
capacities of man. One places a man in a job that suits 
him; the other alters the job to suit the man; neither 
attempts to alter or control man’s behavior. 

For years in the laboratory we have controlled the 
behavior of experimental subjects—both animal and hu- 
man—by a widening array of principles and techniques. 
The new technology of education is the application of 
behavioral laws in modifying or controlling behavior. 
Such a technology became possible with the realization 
that we are actually referring to a verbal repertoire 
(Skinner, 1957) controlled by the same laws as other 
behavior. The old, defunct explanatory concepts of 
knowledge, meaning, mind or symbolic processes have 
never offered the possibility of manipulation or control; 
but behavior, verbal or otherwise, can be controlled with 
ease and precision. 

While machines are not the essential or defining as- 
pects of this technology, they do play an important role 
in providing some of this fine control the technology re- 
quires. We will now examine several machines and notice 
the advantages they offer. 

At Harvard there ‘is a self-instruction room with ten 
booths, each containing a machine. The student gets one 
set of material from the attendant and places it in the 
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machines. He closes the machine and begins his studies. 

This machine presents one item of material at a time. 
The subject reads the statement which has one or more 
words missing and he completes the statement by writing 
in the answer space. He then raises the lever and a small 
shutter opens revealing the correct answer, and simultane- 
ously his answer is moved under glass where it can be 
tead and compared with the new-exposed correct answer. 
After comparing his answer with the correct answer the 
student indicates to the machine, with an appropriate 
movement of the lever, whether his answer was correct 
or incorrect and the next item appears in the window. 
All items answered wrong are repeated after he completes 
the set of items. Correctly answered items are not re- 
peated. 

A critical feature of the machine is that it provides 
immediate reinforcement for correct answers. Being cor- 
tect is known to be a reinforcer for humans (Perin, 
1943). In machine teaching reinforcement is immediate. 
‘We know from laboratory work that a delay between a 
response and its reinforcement of a few seconds will 
greatly reduce the effectiveness of the reinforcement. 
Adult humans have developed behavior which serves to 
mediate small delays; nevertheless, any delay makes rein- 
forcement less effective. 

Although other techniques such as programmed work- 
books (Homme and Glasser, 1958), “self-correcting” 
homework (Diederich, 1960), and flash cards are some- 
times used in this new behavioral technology, they offer 
less control. Teaching machines eliminate undesirable 
forms of responses which would also be successful in 
obtaining the right answer. For example, the teaching 
machine insures that the student answer before peeking 
at the indicated answer. There is a strong temptation to 
glance ahead with,only a poorly formulated, unwritten 
answer when flashcards are used. 

This write-in machine is a prototype of the most com- 
mon machine. There is another machine used for teach- 
ing young children material which consistently has a 
single possible answer. In the machine the constructed 
answer is automatically compared with the true answer. 
The child is presented a problem, pethaps a statement 
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such as 2+2= , and he must provide the “4.” 
By moving a slider appropriately he can insert the 4 into 
the answer space. He then turns the crank, and the next 
item appears immediately, and therefore, immediate re- 
inforcement is provided. _ 

Both of the machines described thus far require the 
student to compose the answer. A machine for a less 
mature organism who cannot yet compose an answer can 
be used for teaching preschool children (Hively, 1960). 
In this machine there is a large top window and three 
small windows. In the large window there is some sort 
of problem, and in the three smaller windows, there are 
three alternative choices. When the correct choice is 
made, the next frame is presented. 

Enough of machines. They should not be allowed to 
obscure the truly important feature of the new tech- 
nology, namely, the application of methods for behav- 
ioral control in developing programs for teaching. We 
need to say no more about the well-known principle of 
immediate reinforcement. Our second principle is also 
well known. Behavior is learned only when it is emitted 
and reinforced. But in the classroom, the student per- 
forms very little, verbally. However, while working with 
a machine, the student necessarily emits appropriate be- 
havior and this behavior is usually reinforced since the 
material is designed so that the student is usually correct. 
Not only is reinforcement needed for learning, a high 
density of correct items is necessary because material 
which generates errors is punishing. Laboratory experi- 
ments (Azrin, 1956) have shown that punishment lowers 
the rate of the punished behavior. In our experience with 
teaching machines we have also observed that students 
stop work when the material is so difficult that they make 
many errors. Furthermore, they become irritated, almost 
aggressive, when errors are made. ə 

The third important principle is that of gradual pro- 
gression to establish complex repertoires. A visitor once 
asked if Skinner had realized that pigeons were so smart 
before he began using them as subjects. The answer given 
by a helpful graduate “student was that they weren’t so 
smart before Skinner began using them. And indeed they 
weren’t. The behavior developed in many experiments 
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is like that developed in the classroom. Both are complex 
operants. Both require a careful program of gradual pro- 
gression. We cannot wait for a student to describe the 
content of a psychology course and then reinforce the 
performance; nor can we wait for a pigeon to emit such 
an improbable bit of behavior as turning a circle, facing 
a disk on the wall, pecking it if lit, and then bending 
down to a new exposed food tray and eating. 

When developing a complex performance in a pigeon, 
we may first reinforce simply the behavior of approaching 
the food tray when it is presented with a loud click. Later 
the pigeon learns to peck a key which produces the click 
and the food tray. Still later, he may learn to peck this 
key only when it is lit, the peck being followed by the 
loud click and approach to the food tray. In the next 
step, he may learn to raise his head or hop from one foot 
to another, or walk a figure eight, in order to produce the 
lighted key which he then pecks; the click follows; and 
he approaches the food tray. 

This principle of gradual progression runs through 
many of the teaching machine techniques. Both human 
and avian scholars deserve the same careful tutorage. The 
teaching machine program moves in very finely graded 
steps, working from simple to an ever higher level of com- 
plexity. Such a gradual development is illustrated in 
Table 1 by a few items taken from a psychology program 
(Holland and Skinner). 

The principle of gradual progression serves not simply 
to make the student correct as often as possible, but it 
is also the fastest way to develop a complex repertoire. 
In fact, a new complex operant may never appear except 
through separately reinforcing members gf a graded series 
(Keller and Schoenfield, 1950). Only this way can we 
quickly create a new pattern of behavior. The pigeon 
would not have learned the complex sequence necessary 
to receive the food if he had not learned each step in its 
proper order. Obviously a child can’t begin with advanced 
mathematics, but neither can he begin witl 2 + 2 = 4— 
even this is too complex and requires a gradual progres- 
sion. , 

Our fourth principle is, in a sense, another form of 
gradual progression—one which involves the gradual 
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TABLE 1 


Items from the psychology program (Holland and Skinner). 
These items illustrate the gradual development of a new concept. 


Item 


. Performing animals are some- 


times trained with “rewards.” 
The behavior of a hungry ani- 
mal can be “rewarded” with 


. A technical term for “reward” 


is reinforcement. To “reward” 
an organism with food is to 
it with food. 


organism can be ..... 
with water. 


A school teacher is likely, 
whenever possible, to dismiss 
a class when her students are 
rowdy because she has been 
AE N by elimination 
of the stimuli arising from a 
rowdy class. 


The teacher who dismisses a 
class when it is rowdy causes 
the frequency of future rowdy 
behavior to (1) ...--+- +++ $ 
since dismissal from class is 
probably a(n) (2) 
for rowdy children. 


If an airplane spotter never 
sees the kind of plane he is to 
spot, his frequency of scanning 
the sky (1) ...---- onde In 
other words, his “looking” 
behavior is (2) 


Percentage 
of Students 
Giving 

Correct Answer the Answer 
Food 96 
Reinforce 100 
Reinforced 100 
Reinforced 92 
(1) Increase 86 


(2) Reinforcement 


(1) Decreases 94 
(2) Extinguished 

(or: Not 

Reinforced) 
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withdrawal of stimulus support. This we shall call fading. 
This method will be illustrated with some neuroanatomy 
material.1 A fully labelled cross section of the medulla 
oblongata is placed before the student while he works 
with a large set of items pertaining to the spatial arrange- 
ment of the various structures. For example, “posterior 
to the cuneate nuclei are the —___.” The answer is: 
“the cuncate fasciculi.” 

After many such items, he begins another set and has 
another picture before him, but now the structures are 
only labelled with initials. A new set of items again asks 
a long series of questions pertaining to the spatial posi- 
tion of the various structures. For example, “between the 
gracile and the trigeminal nuclei are —__—.”” The an- 
swer is the “cuneate nuclei.” 

After many more items, he proceeds to a new set and 
the next picture. This time the picture is unlabelled. 
Again, he goes through a series of new items, not simple 
repetition of the previous ones, but pertaining to the 
Same program of the spatial location of the different 
structures. This set is followed by still another, but with 
no picture at all. He is now able to discuss the spatial 
position of the structures without any visual representa- 
tions of the structures before him. In a sense, he has his 
own private map of the medulla. He may further demon- 
strate his newly acquired ability by accurately drawing the 
medulla. 

The neuroanatomy example is an elaborate example of 
fading. Fading is also applied in a more simple form in 
constructing verbal programs without pictorial displays. A 
single item may in one sentence give a definition or a 
general law and a second sentence in that same item an 
example in which a key word is omitted. This would be 
followed by a new example in the next frame, but with 
the definition or law lacking. 

This brings us ‘to our fifth principle, control of the 
student’s observing and echoic behavior, In the classroom 
the student is often treated as though he were some kind 

* This material was prepared by D. M. Brethower in col- 


laboration with the present author and is being used at Harvard 
for research purposes. 
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of passive receiver of information who can sop up infor- 
mation spoken by the teacher, written on the blackboard, 
or presented by films. But all of these are effective only 
insofar as the student has some behavior with respect to 
the material. He must listen carefully, or read carefully, 
thus engaging in usually covert echoic behavior. 

Ineffectiveness of classroom techniques is often cred- 
ited to “inattention” or poor “concentration.” It has 
been shown (Reid, 1953) that if a discrimination is to 
be learned, adequate observing behavior must first be 
established. We have further found (Holland, 1958) 
that observing behavior, or, speaking loosely, “attention” 
is subject to the same forms of control as other behavior. 
This control of observing behavior is of prime impor- 
tance. When the student becomes very “inattentive” in 
the classroom, the teaching material flows on; but with 
a machine, he moves ahead only as he finishes an item. 
Lapses in active participation result in nothing more 
than the machine sitting idle until the student con- 
tinues. 

There is, however, a more subtle aspect to the control 
of observing behavior than this obvious mechanical one. 
In many of the examples we have seen, success in an- 
swering the problem only depends on the student’s care- 
ful observation of the material in front of him at the 
moment. This may be illustrated by more material from 
the psychology program. A graph showing stimulus gen- 
eralization data is in front of the student while he works 
on the machine. In the program he may complete a state- 
ment like the following: “As the wave length changes in 
either direction from the wave length present during Te- 
inforcement, the number of responses ey ine 
answer is “decreases.” The item serves only to control the 
behavior of observing the data. Of course, many more 
such items are used to discuss the same data. 

This principle of controlled observation extends to the 
details of writing a single item. For example, “Two 
events may have a common effect. An operant reinforced 
with two reinforcers appropriate to different deprivations 
will vary with ———~ deprivations.” The answer is “two 
or “both.” Here the programmer’s choice of the omission 
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serves to insure a careful reading of the item. Only those 
parts of an item which must be read to correctly com- 
plete a blank can safely be assumed to be learned. 

Our sixth principle deals with discrimination training. 
In learning spoken languages, for example, it is necessary 
to be able to identify the speech sounds. A student may 
listen to a pair of words on a special phonograph which 
repeats the passage as many times as he desires. The 
visual write-in machine instructs him to listen to a spe- 
cific passage. For example, the student may hear two 
words such as: “sit, set.” He listens as many times as he 
needs, then writes the phonetic symbols in the write-in 
machine and operates the machine, thereby exposing the 
true answer and providing immediate reinforcement for 
his correct discrimination. 

However, little academic education is simple discrimi- 
nation. More often, it is abstraction or concept forma- 
tion. An abstraction is a response to a single isolated 
property of a stimulus. Such a property cannot exist 
alone. Redness is an abstraction. Anything that is red 
has other properties as well—size, shape, position in 
space, to name a few. There are red balls, red cars, red 
walls. The term “red” applies to all of them but not to 
green balls, blue cars, or yellow walls. To establish an 
abstraction (Hovland, 1953), we must provide many ex- 
amples; each must have the common property, but 
among the various examples there must be a wide range 
of other properties. This is best illustrated by examples 
from a preverbal machine. 

These examples are from a program? which teaches a 
child to respond to the abstract property of form. In 
each item, the upper figure is the sample and the lower 
three are the alternatives. While developing a program 
for establishing an abstraction we remember our earlier 
principles, and move through a gradual progression. For 
the first several items there is a sample and a single 
match, the other two being blank. The sample and its 
match are exactly alike at this stage. ” 

After many such items, we would begin to have others 
in which the sample and its match, again correspond in 
size, color and form, but an additional incorrect alterna- 


* The program was prepared by B. F. Skinner. 
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tive has been added which differs from the sample in 
all these aspects. Later we move on to frames with three 
choices, again the sample and its match corresponding 
exactly. Next, the sample and the match may differ in 
some property such as color or size. It is essential that 
the program contain many items among which the sam- 
ple and correct match differ in all properties except the 
one providing the basis for the abstraction. Otherwise the 
abstraction will be incomplete because the extraneous 
property will share some of the control over the abstract 
response. 

Às we move on with additional examples, the sample 
and the correct match differ both in color and in size, and 
the incorrect alternatives are beginning to share some of 
the extraneous properties with the sample. The student 
continues with many such problems in which the only 
common property between the sample and the correct 
match is the shape, regardless of size and color. Even 
now our abstraction may be incomplete. We have kept 
the figures in only one orientation. Therefore, we also 
have a series in which the samples are rotated. 

A great deal of academic education consists of trying 
to teach abstractions. Concepts such as force, reinforce- 
ment, supply-and-demand, freedom, and many, many 
other possible examples are all abstractions. Furthermore, 
in the academic setting, the student seldom adequately 
forms abstractions. The trigonometry student commonly 
uses triangles with the right angle as one of the two lower 
angles. If the triangle is rotated 90°, so that the right 
angle is upward, the student often does not recognize it 
as a right triangle. Neither is an abstraction developed 
simply by learning a definition. The psychology student 
who learns the definition of reinforcement in formal 
terms and is acquainted with a laboratory example of 
food reinforcement, may not realize the horrible conse- 
quences of sending his girl friend flowers to end an argu- 
ment. Thus, in the psychology program, to develop a 
new concept} we follow the pattern in the preverbal ex- 
ample. A wide range of examples are analyzed which 
differ in as many aspects as possible, each still having the 
common property which characterizes the concept. 

The last principle I shall discuss is really a question of 
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methodology which has served so well in the laboratory. 
This principle is to let the student write the program. A 
few years ago, a cartoon was published in the Columbia 
Jester. The rat leaning on a bar is saying to the other rat: 
“Boy, do we have this guy conditioned. Every time I 
press the bar down, he drops a pellet in.” 

Although said in jest, it is true that the rat controls 
the experimenter’s behavior. When interesting things are 
observed about the rat’s behavior, the control circuits are 
rewired to investigate the interesting new facet of behav- 
ior. In a sense, the rat is wiring the control circuit. Simi- 
larly, the behavioral engineer who prepares good teaching 
machine material must be under the control of the stu- 
dent’s responses. When the student has trouble with part 
of a program the programmer must correct this. The stu- 
dent’s answers reveal ambiguities in items; they reveal 
gaps in the program and erroneous assumptions as to the 
student’s background. The answer will show when the 
program is progressing too rapidly, when additional 
prompts are necessary, or when the programmer should 
try new techniques. When unexpected errors are made, 
they indicate deficiencies not in the student but in the 
program. 

The most extensive experience with this principle of 
modifying the program to fit the student has been at 
Harvard (Holland, 1959) with the psychology program. 
In 1958, we had a program consisting of forty-eight disks 
or lessons of twenty-nine frames each. After using the 
program and making a detailed, item-by-item analysis of 
the students’ answers we diagnosed the particular defici- 
encies in the program and revised it accordingly. The 
program was also extended to cover a larger amount of 
subject matter and, in 1959, it consisted of sixty disks. 
You have already seen a few items from the course. After 
using the revised material in 1959, we evaluated the ex- 
tent of its improvement. Table 2 shows the percentage 
of errors on the first twenty disks for each of the two 
years. ‘ 

The revision eliminated about half the errors. The last 
column of the table gives percentage of improper self- 
scoring by the students. Revision also cut these scoring 
errors approximately in half. Furthermore, the revision 
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TABLE 2 
A comparison of the student's errors in 
using the revised (1 959) and tinrevised 
(1958) program in psychology. 


Per cent Items 
Per cent Improperly Scored 
Errors by Students 
1958 20.1 3.6 
1959 11.0 14 


decreased the time required to complete the material. 
Although the second year’s material had more disks— 
sixty as opposed to forty-eight—it actually required the 
average student about one hour less to complete the 
work than it had required for the shorter first version. 
Frequency distributions on the median times in minutes 


for completion of the various disks are shown in Fig. 1. 


35 — 1959 
---— 1958 


25 


PERCENTAGES OF DISKS 
a 


65 05 145 185 225 26.5 30.5 
MEDIAN TIMES OF ALL CYCLES (MINUTES) 


Fig. 1. Here are the frequency distributions for the median 
times to complete disks or “lessons” for ghe revised and unre- 
vised psychology program. Raw frequencies were converted to 
percentages to equate the area under the curves. 

° 


These are the times required for the median student to 
move through each set of material answering every item 
once and to repeat the items answered incorrectly. Notice 
the considerable time required for many disks in the first 
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year’s material. Primarily this was because students re- 
peated the larger number of items missed in the first 
cycle. 

"But the improved material provided faster perform- 
ance, even when the delay due to repetition of incorrectly 
answered items is not considered. The frequency dis- 
tributions for the first cycle only are provided in Fig. 2. 


PERCENTAGES OF DISKS 


6.5 10.5 14.5 18.5 225 


MEDIAN TIMES OF FIRST CYCLES (MINUTES) 


_ Fig. 2. Here are the frequency distributions for the median 
times to complete only the first cycles for the revised and unre- 
vised psychology program. Raw frequencies were converted to 
percentages to equate area under the curves. 


These data exclude the time used in repeating items. 
Here too the revision produced slightly more rapid prog- 
ress. 

_ Such careful tailoring of material to fit the student is 
impossible with most teaching techniques. With teaching 
machines, as in no other teaching technique, the pro- 
grammer is able to revise his material in view of the 
students’ particular difficulties. The student can write 
the program; he cannot write the textbook. 

We have seen that the principles evolved from the 
laboratory study of behavior have provided the possibility 
for the behavioral engineering of teaching. This new 
technology is thoroughly grounded in some of the better 
established facts of behavioral control. The future of 
education is bright if persons who prepare teaching ma- 
chine programs appreciate this, and appropriately educate 
themselves in a special, but truly not esoteric, discipline. 
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But it is vital that we continue to apply these techniques 
in preparing programs. The ill-advised efforts of some of 
our friends, who automatize their courses without adopt- 
ing the new technology, have an extremely good chance 
of burying the whole movement in an avalanche of 
teaching-machine tapes. 
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Skinner’s Theory of 
Teaching Machines* 


Davin ZEAMAN 
University of Connecticut 


One of the differences between the natural and the 
social sciences is that in the natural sciences, each suc- 
ceeding generation stands on the shoulders of those that 
have gone before, while in the social sciences, each gen- 
eration steps in the faces of its predecessors. 

Whether psychology is a natural science or an un- 
natural one may be a question, but given a practical psy- 
chological problem of training devices to solve, there is 
no question at all that we have a choice of two strategies 
here, the shoulder or the face. 

If it is the shoulder, where should the constructor and 
the programmer of human training devices look in the 
scientific literature of psychology for experimental data 
relevant to his task? 

Any answer to this question implies a theory of train- 
ing devices, It is in this loose but yet important sense 
that the word theory is here used. 

Skinner has a theory, if I read him gightly, and it says 
the appropriate reference class of experiments, the ap- 
propriate experimental paradigm, for construction and 

2 


* This article originall appeared in Eugene Galanter (Ed.), 
Automatic Teaching: The State of the Art. New York: John 
Wiley & Sons, Inc., 1959, Chapter 14. It is used here with the 
permission of the author and the publisher. 
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programming teaching machines is that of free operant 
conditioning. Others, in particular those using the mem- 
ory drum, are held to be inappropriate. ; 

Skinner’s resolution of the shoulder-face question is, 
happily, the shoulder, but the shoulders are too ivy- 
league-ish—too narrow. 

It is the counter-thesis of this talk, that existing teach- 
ing machines (and those likely soon to be built) repre- 
sent complex admixtures of at least three basic experi- 
mental paradigms, free operant, controlled operant, and 
classical conditioning, and that of these, the free operant 
may be the least pertinent. Furthermore, of all the com- 
monly used apparatuses for experimental study of learn- 
ing that one which most closely resembles one of Skin- 
ner’s teaching machines (the Write-in Model) is the 
memory drum. 

Since there is something less than universal agreement 
on classification of learning paradigms, let me ask for a 
temporary convention in the use of the words “controlled 
operant, free operant, and classical conditioning.” 

All three refer to experimental arrangements of ob- 
servable variables that a machine can handle. The com- 
mon elements of all three include two stimulus events 
S, and S, occurring in that order, and at least two re- 
sponses arbitrarily designated as correct and incorrect. 

By “controlled operant” or “discrete trial operant” is 
meant those experiments incorporating the following ac- 
quisition features: (keeping the Graham-Gagne Runway 
experiments in mind will help you here). 

The acquisition trial begins with the presentation of 
a stimulus situation S, which lasts until the subject makes 
a correct response—R. One and only one correct response 
R occurs per trial because the correct response terminates 
S,. It also brings Sg, which in this case is a reinforcing 
stimulus. S does not elicit R. The timing arrangements 
are partially subject controlled, partially experimenter. 
The time interval between S, and Sy is subject controlled 
since S is contingent upon R, but the intetirial interval, 
that is the Sọ — S, interval is usually under the experi- 
menter’s control. . 

Acquisition is measured by S-R latency or the condi- 
tional probability of R given S,. 


Davin ZEAMAN 51 


_ Illustrative apparatuses for controlled operant condi- 
tioning include the runway, Miller-Mowrer Shuttle Box, 
Retractible Bar Skinner Box, the Single-door Lashley 
Jumping Stand, and also if correction technique is used, 
the Simple T-Maze, Discrimination Boxes and the Two- 
door Jumping Stand. 

_ In the free operant or ordinary Skinner box situation, 
similar features are present with some exceptions. The 
initial stimulus situation S, is of prolonged duration, al- 
lowing many correct responses to occur, each one rein- 
forced during continuous acquisition by the contingent 
reinforcing stimulus Sə. This of course permits a rate 
measure to be taken. 

„Both the free and controlled operant paradigms, then, 
display an SRS sequence, with the second S contingent 
upon R. The big difference lies in the number of correct 
responses allowed per trial, only one in the controlled 
operant (since the correct response eliminated S, as well 
as bringing Sa), but many correct responses in the free 
operant. 

What about timing differences? Is one self-paced and 
the other not? If one accepts Estes’ analysis of the free 
operant, the animal turns S, on and off by himself, even 
though the experimenter leaves it on. Each reinforcing 
stimulus Są turns over the S, and the subject gets a fresh 
look at S}. In this analysis, the free operant is self-paced, 
in the sense that the subject controls the intertrial inter- 
val, the time between Sz. and S4. 

This is to be contrasted with the controlled operant 
in which the subject waits for the experimenter to turn 
on Sj. It is not completely self-paced. _ ; 

It is important to note here that if you accept this 
analysis and regard the free operant as self-paced, it is 
only so when the S,’s remain the same. ; 

If the experimenter has a number of different S,’s to 
be connected to a number of different correct responses, 
free operant techniques can be used, but then it is no 


longer complctely self-paced. The experimenter changes 


the problem. The inter-problem interval is not subject 
] that we are usually in- 


controlled, and it is this interva 


terested in with teaching machines. —. j 
Now finally to the classical conditioning paradigm. 
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The previous two paradigms were SRS sequences with a 
contingency. This one is an SSR sequence with no con- 
tingency. To fulfill the paradigm, the first stimulus S, 
must not elicit R as strongly as you would wish, whereas 
the second stimulus S must elicit R, the correct response. 

This is a highly operational, non-theoretical specifica- 
tion of the classical conditioning paradigm. It is not as- 
serted that Ss be an innate elicitor of R, nor even a strong 
stimulus. Whether such a paradigm always provides the 
necessary or sufficient conditions for learning is an empiri- 
cal question not easily resolved, given the existing litera- 
ture. It is, however, a paradigm that can be used in con- 
struction of training devices, and it looks operationally at 
least to be different from free and controlled operant con- 
ditioning, in not having a contingent Są and in requiring 
S» to elicit the correct response. 

In the matter of timing, classical conditioning resem- 
bles the controlled operant in that the experimenter con- 
trols the inter-trial interval (the Sə, S, gap) but classical 
conditioning differs from both free and controlled oper- 
ants in also giving control over the S, — Sg interval to the 
experimenter. 

With these three paradigms specified in this way, let 
us take a look at three pieces of apparatus, a memory 
drum set up for paired associates, Skinner’s Write-in 
Model teaching machine and Skinner’s Arithmetic ma- 
chine. 

First the paired associates experiment. On the first 
trial around, there is ordinarily no anticipation. The sub- 
ject sees the S, syllable and reads the correct associate. 
This fits the classical conditioning paradigm. It is an SSR 
sequence, no contingency, and all stimulus intervals 
under control of the experimenter. 

On the second and subsequent trials, the possibility 
of operant conditioning enters. If the subject correctly 
anticipates the associate, he is reinforced by the match 
of his anticipated answer and the forced answer. Since 
this matching or knowledge of correct results, is con- 
tingent upon a correct anticipation, elements of a con- 
trolled operant are present. $ 

If the subject gives the wrong response, the contingent 
reinforcement is not present, and the trial resembles an 


Davin ZEAMAN 53 


extinction trial of a controlled operant. The trial ends, of 
course, with a classical conditioning feature, a forcing of 
the correct response, elicited by the appearance of the 
associate. 

If no overt response is given at all, the trial becomes a 
simple classical conditioning trial again. 

Thus there are three kinds of trials possible with a 
paired associates machine, a simple classical trial, a con- 
trolled operant acquisition trial with a terminal classical 
feature, or a controlled operant extinction trial with a 
terminal classical feature. The timing of these trials is 
almost exclusively that of a classical conditioning, how- 
ever both the S} — So interval and the Sa — S, interval are 
experimenter controlled. To my knowledge there are no 
animal experiments with trials at all like these, although 
I believe they might be arranged. 

Consider now Skinner’s Write-in Model learning ma- 
chine. It presents a written question or an incompleted 
sentence or paragraph in an aperture, and then waits for 
the subject to write in his answer or completion. Follow- 
ing this the subject operates the machine to present the 
correct written answer to be compared with his own. The 
next problem appears when the subject indicates to the 
machine that he is ready. 

We analyze the machine in the terms used before. As 
with the memory drum there are three kinds of trials, 
depending upon whether the subject makes no response, 
makes the correct response, or makes an incorrect re- 
sponse. 7 : 

If no response is made, the trial looks like a classical 
conditioning paradigm or SSR. The question stimulus 
S, is presented. Then the answer stimulus follows, and 
elicits the answer response. No contingency, just a forced 
response like classical conditioning. 

If the correct response is made, an instrumental or 
operant feature emerges. The correct answer given in 
anticipation of Sẹ is reinforced by a matching of the 
correct results» This reinforcement is of course contin- 
gent, and makes the trial resemble the free or controlled 
operant, But since there is only one correct response 
made, this aspect of the trial looks most like the con- 


trolled operant. 
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If an incorrect response is made, the contingent match- 
ing reinforcement is absent and we have elements of a 
controlled operant extinction trial. The trial is terminated 
however with the forced response characteristic of the 
classical procedure. 

The high similarity of these three kinds of trials with 
those of the paired associates memory drum experiment 
is obvious. Yet there is some difference. A difference in 
timing. With the usual paired associates procedure, all 
stimulus intervals are experimenter controlled. With ‘the 
Write-in Model, in contrast, the S, — Sg interval is sub- 
ject controlled (that is, the subject can give himself as 
much time as he wants before presenting Sẹ) and the 
interproblem interval is also subject controlled. It is in- 
deed a self-paced device. 

The subject’s control of the timing of the S, stimulus 
is by no means new: it is a property of free and controlled 
operant procedures in general, but the subject’s control 
of the interproblem interval is relatively novel. It is not 
found in either the usual controlled operant experiment, 
or in the usual free operant, nor of course, in classical 
conditioning. Some recent experiments on the role of 
observing responses in animal discrimination have incor- 
porated an analogous type of paradigm. Wyckoff had 
pigeons turning on their own discriminative stimuli by 
stepping on a treadle. His work should be relevant to 
this machine. And other work may be relevant too. 

Since there have been so many memory drum experi- 
ments done, and since it is no engineering trick at all 
to convert the usual memory drum to an entirely self- 
paced apparatus, like Skinner's Write-in Model, I would 
be surprised if a literature survey failed to turn up any 
self-paced paired associates data. Of course when a sub- 


ject riffles through a deck of flash cards at his own time, » 


the paradigm for this primitive machine is the same as 
that of Skinner’s Write-in Model. 

The last device to be inspected is a modification of one 
of Skinner's devices which we have also constructed and 
called the “Arithmachine.” It presents visual material 
and allows the subject to make any of a large number of 
numerical responses, one of which is correct. The re- 
sponse is the movement of sliders and the turning of a 
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crank. If the sliders have been positioned correctly, each 
slider position representing a number, the turning of the 
crank will bring reinforcement and the next problem. 

The analysis. This machine operation looks much more 
like the controlled operant procedure than does the 
Write-in Model. The S, or question is presented, a re- 
sponse is waited for, and a contingency set up. If the 
response is correct, the subject is rewarded by the com- 
plete turning of the crank and whatever other reinforcing 
signals the experimenter wants. There are no forced re- 
sponses of a classical nature at the end of the trial. The 
correct response not only brings reinforcement, it elimi- 
nates Sı, the question stimulus, thus allowing only one 
correct response per trial, the hallmark of the controlled 
operant. 

Timing is not completely self-paced. The S, — So inter- 
val is subject controlled, but the interproblem interval is 
not. The problems come in massed sequence. As soon 
as the subject finds out that he has been right on the last 
problem, the new one stares up at him. 

Strength of acquisition is inferred with this apparatus 
from latency or probability of response, as it is for the 
other machines mentioned: Free rate of operant response, 
in the presence of a prolonged S; is not taken here, nor is 
it a measure taken for any teaching machine that I know 
of. 

This machine, in summary, embodies all of the ele- 
ments of the controlled operant, and the literature on 
this kind of experiment may therefore be relevant. 

In final summary, the original question is te-asked, 
where should the constructor and programmer of human 
training devices look in the scientific literature of psy- 
chology for experimental data relevant to his task. Any 
answer to this question implies a theory of training de- 
vices. If Skinner’s theory says that the literature on free 
operant is most relevant, this paper proposes the counter 
theory that the literature of controlled operant and classi- 
cal conditioning is also relevant but more so. 

If Skinner’s theory says that memory drum experi- 
ments provide an inappropriate model for construction 
of training devices, this paper proposes the counter theory 
that no other experimental device or procedure more 


closely resembles Skinners Write-in Machine than a 
common memory drum paradigm. 

Old Ebbinghaus’s shoulders may not be ivy-league, but 
we ought to stand on them anyway. 


PARADIGMS 
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A Review of the Literature on 
Certain Aspects of Automated 
Instruction* 


W. J. Carr 
Temple University 


The primary purpose of this paper is to review the fast- 
growing literature on the design and utility of automatic 
teaching devices. A secondary purpose of the paper is to 
suggest a number of hypotheses, the testing of which 
might further our understanding of the variables which 
influence the effectiveness of automated instruction in 
the classroom. s 

The plan of the present paper is as follows: First, there 
will be an attempt to provide a conceptual framework in 
which to place a discussion of automatic teaching de- 
vices. Second, a review will be made of the rather exten- 
sive literature on automated instruction. In this section, 
the writer will be concerned with two types of studies, 


*A report submitted (January 1959) to Dr. J. Charles 
Jones, Director, Susquehanna Valley Program, Bucknell Uni- 
versity. Support for this work was received from a grant from 
the Fund for the Advancement of Education to Bucknell 
University. Thè report is used here with the permission of the 
author, Similar versions of this paper have appeared in 
Lumsdaine, A. A., and Glaser, Robert (Ed.), Teaching Ma- 
chines and Programmed Learning, Washington, D.C.: National 
Education Association, 1960, and WADC Technical Report 
aoe: Air Research and Development Command (August 

59). 
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(a) simple evaluation studies, in which the researcher 
compared the effectiveness of a given teaching device 
with the “standard” form of instruction and (b) para- 
metric studies, in which the researcher investigated one or 
more variables of which the effectiveness of the teaching 
device might be a function. Finally, there will be an 
attempt to summarize some of the current thinking as 
to how research in this area might best proceed and to 
suggest some specific hypotheses. 


SOME PRINCIPLES OF LEARNING 


For almost one hundred years, psychologists have ex- 
pended a considerable portion of their research effort 
toward an understanding of the learning process, chiefly 
by studying the variables which influence the rate of 
learning and forgetting. Out of this effort has come a 
number of reliable principles of learning which can be 
utilized to design an efficient learning situation. 

Gagne (1958) has reviewed the literature on learning, 
on a very broad scale. His paper encompasses almost all 
of the known variables which influence learning and 
retention. Moreover, Gagne has discussed at some length 
the importance of using adequate measures of learning, 
which necessarily includes the problem of transfer from 
the classroom to real life. 

Skinner (1954) and Gilbert (1958) have summarized 
those principles of learning which have led a number of 
workers to consider seriously the development of auto- 
matic teaching devices for use in the classroom. A re- 
phrasing of these principles, couched in non-technical 
terms, might take the following form. Accompanying 
each principle is a brief statement showing how that 
principle is sometimes violated in the typical classroom. 


1. Learning takes place most rapidly if the student is 
actively engaged with the subject matter. At the outset 
of training, some time might profitably* be spent in 
watching or listening to someone else perform the acts 
to be learned, but the student will become proficient 
only if he practices the acts himself. This principle ap- 
plies to verbal skills as well as to non-verbal ones. Unfor- 
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tunately, in most classrooms the leaming of most subject 
matter is thought of as being a spectator, rather than a 
participator sport. 

2. Learning is most effective if the student develops 
the skills and knowledge in a form which will readily 
generalize to the “real life” situation for which they are 
intended. Usually, this means that the student must learn 
to construct correct answers to questions, rather than 
merely be able to recognize them. Thus, the responses 
which the student makes in the learning situation should 
be of the former rather than of the latter type. All too 
frequently, in the classroom of today, recognition is used 
as a criterion of learning. 

3. Learning takes place most rapidly if immediate 
“knowledge of results” 1 is given for each response. If the 
response is correct, the student should be so informed, 
just as soon as the response is completed. If the response 
is incorrect, he should know this, also. Even a slight delay 
drastically retards learning. Presently, our students must 
frequently endure long delays before they receive feed- 
back of this sort. 

4. Learning takes place most rapidly if the subject 
matter is organized in a hierarchic form. Lessons should 
begin with very simple problems, for which the student 
already has the necessary skills and knowledge. Gradually, 
the difficulty level may be raised until the desired level 
of proficiency is obtained. Care must be exercised so that 
at each point the student is given new information to 
be used to solve the next problem. Ideally, the subject 
matter should be presented in steps so small that the 
success of the student on the next problem is practically 
guaranteed. Unfortunately, both textbooks and classroom 
techniques frequently ignore this principle. p 

5. Receiving frequent “knowledge of results” keeps 
students working at the assigned task. The hurdles which 
the student must surmount should be kept immediately 


2 The sophisticated reader will recognize that mere “knowl- 
edge of results” does not guarantee rapid, learning. Strictly 
speaking, learning takes place only if the “knowledge of re- 
sults” also constitutes reinforcement. A reinforcement is a 
stimulus which increases the probability of the stimulus- 
response connection which it follows. 
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before him and by making successes frequent, the student 
will develop a high level of aspiration. In the typical 
classroom, the student is studying to prepare himself for 
a far-off quiz or test. The result is that the teacher must 
spend a considerable portion of her time devising some 
artificial motivators, to keep the students working. 

6. Since learning takes place in individuals, the learn- 
ing situation should be designed so that each student 
may proceed at his own pace. For various reasons, some 
students learn faster than others and therefore, it is diffi- 
cult to have them learn in groups.? This principle does 
not necessarily dictate that students must work alone. It 
merely emphasizes the belief that if students are to learn 
in groups, the learning situation should be designed so 
that the fast learners are not impeded by the slow ones 
and the slow learners are not confused by too stiff a pace. 


The list of principles given above is neither new nor 
exhaustive. Good teachers have known of theses princi- 
ples for a long time and have endeavored to utilize them 
as best they can, within the limitations of the present-day 
classroom situation. 

But it must be recognized that the limitations imposed 
on the teacher by the present-day classroom situation are 
extremely severe. Many educators and psychologists have 
been impressed with the folly of trying to teach thirty or 
more children, simultaneously. Consider, for example, the 
teaching of the multiplication tables. While one student 
recites, the teacher is able to reinforce each of his re- 
sponses. But the other children are merely listening and 
probably little learning occurs. Further, if the teacher 
uses the weekly quiz as a learning device, learning suffers 
from the long delay between the responses and the 
knowledge of results. Finally, if the class is working on 
the “five” tables, some bright students might more 
profitably be work'ng on the “sevens” while some of the 
slower ones should still be working on the “threes.” True, 
making the class more homogeneous with respect to 
mathematical ability might help, but this poses special 
administrative problems. 

* The writer recognizes that the development of some skills 


Tequires the presence of a group. In this discussion we have 
teference to those skills and knowledge which do not. 
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The teacher’s inability to work efficiently with more 
than a very few studetns at a time has caused a number 
of psychologists to follow the lead of Sidney L. Pressey 
and B. F. Skinner in recommending a thorough inves- 
tigation of teaching by means of automatic teaching 
devices. Presumably, each student would have the daily 
use of a machine for a period of time. Theoretically, at 
least, the machine could be designed so that its opera- 
tion would be congruent with all of the principles of 
efficient learning which have been here reviewed. 

Skinner (1958) has discussed some of the social and 
economic ramifications of his proposal to make wide use 
of automatic teaching devices in our educational system. 
Space will not be devoted to this topic in the present 
paper, except to point out that few if any workers in this 
field intend to build devices designed to replace the 
teacher in the classroom. At the very best, the devices can 
only hope to supplement the efforts of the teacher, by 
relieving her of the task at which she is probably least 
capable—delivering “knowledge of results” rapidly to her 
students for each response directed toward the subject 
matter. If effective, the use of such devices would free the 
teacher for tasks which she can perform well and for 
which no machine can replace her. 


A REVIEW OF LITERATURE 


Porter (1958a) has recently published an excellent 
review of a portion of the extensive literature dealing 
with the many devices which have been suggested as 
being useful adjuncts to the teaching process. Further, 
he has developed a system of classification of these de- 
vices which promises to be quite useful. He has distin- 
guished among three classes of devices: 

Stimulus Devices: Stimulus devices are those which 
present some information to the learner, by way of one 
or more sense modalities. Frequently, they show how ot 
under what conditions something is to be done. Exam- 
ples of this class of device are: models, projecting devices, 
phonographs, braille, and the like. It should be pointed 
out that while these devices do present subject matter to 
the learner, they do not necessarily require any action on 
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his part and thus learning is not assured. For this reason, 
Porter suggests that they be thought of as teaching aids, 
rather than teaching devices. 


Response Devices: Response devices are those which 
provide the learner with an opportunity to practice some 
activity. These devices are usually used to collect or ma- 
nipulate data. Examples of this class of device are type- 
writers and desk calculators (as data manipulators) and 
the Classroom Communicator or the GSR recorder (as 
data collectors). Thus, a response device permits the 
learner to practice some activity, but it does not neces- 
sarily provide him with subject matter on which to prac- 
tice. For this reason, Porter suggests that response devices 
also be thought of as teaching aids, rather than as teach- 
ing devices. 


Stimulus-Response Devices: Stimulus-Response devices 
are those which both present information to the learner 
and also require some appropriate action on his part. 
Further, such devices usually respond differentially, de- 
pending upon the behavior of the learner, Examples of 
the stimulus-response class of device are the Pressey 
multiple-choice apparatus, Skinner’s devices and the clas- 
sical memory drum. It should be noted that the stimulus- 
response device more nearly simulates the role of the 
teacher by both presenting information to the learner 
and requiring some appropriate action on his part. Porter 
feels that these might properly be called teaching de- 
vices, since no human teacher is required to mediate 
relations between the machine and the learner, except 
to service or maintain the machine. 


Further, Porter distinguished among three types of 
stimulus-response devices. First, there is the simulator, 
which duplicates the essential characteristics of some 
complex task and which Tequires appropriate action from 
time to time on the part of the learner. Ordinarily, this 
type of device does not provide immediate feedback. The 
electronic flight simulator is a good illustiation of this 
type of device. 

The second type of stimulus-response device distin- 
guished by Porter is the immediate reinforcer, which not 
only presents the learner with a problem situation but 
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also provides him with immediate “knowledge of results” 
concerning his answers. It should also be noted that this 
type of device is learner-paced, in that the machine’s 
reaction is contingent upon the learner’s behavior. Pres- 
sey’s multiple-choice device and Skinner’s teaching ma- 
chine are examples of the immediate reinforcer. 

The third type of stimulus-response device is what 
Porter called the pacer. A pacer is a device which pre- 
sents stimulus materials for a given interval of time and 
then provides the appropriate response, whether or not 
the learner has attempted to answer. The memory drum, 
so frequently used in psychological research on learning 
phenomena, illustrates this type of device. 

Of the three types of stimulus-response devices distin- 

guished above, the one which seems to offer the most 
promise for use in the class is the immediate reinforcer. 
Simulators ordinarily do not provide immediate feedback 
and the operation of the pacer is independent of the 
behavior of the learner. Thus, these latter types of de- 
vices seem to violate one or more of the principles of 
learning referred to earlier. Hereafter, in the present pa- 
per, consideration will be given only to those stimulus- 
response devices which are of the immediate reinforcer 
type. 
vi word of caution might be added with respect to 
Porter's classification of learning machines. It is quite 
possible that a slight modification of a stimulus-device or 
a response-device might enable it to qualify as a stimulus- 
response device. Also, it is possible for both simulators 
and pacers to be reclassified as immediate reinforcers, if 
appropriate changes are introduced. Porter has classified 
particular devices in accordance with the design of the 
machine offered by its designer. 


SOME HISTORY OF AUTOMATED INSTRUCTION 


Perhaps the first attempt to develop an automatic 
teaching device was made in 1918 by English (1942). 
This worker constructed and tested a device by which 
soldiers rapidly learned to squeeze rather than jerk a 
rifle trigger. Knowledge of results was delivered to the 
soldier via a manometer attached to the trigger. However, 
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it should be pointed out that the subject matter in this 
case was quite simple, consisting of a single habit. 

Sidney Pressey (1950) is often credited with being the 
first to give serious thought to the widespread use of 
teaching devices in the classroom. Although, at the outset, 
Pressey was primarily interested in automatic testing 
devices, he soon discovered that students could learn by 
taking tests via his machines. Pressey and his students 
have published a good number of papers in which various 
designs of machine have been described and evaluated. 

Basically, the type of device developed by Pressey 
follows the format of the multiple-choice question. The 
student is presented with a question and four options, 
only one of which is correct. He chooses an option and 
the device provides some immediate feedback. If the 
student has chosen the correct answer, the machine 
presents the next question. If an incorrect option is 
chosen, the student must continue responding until he 
selects the correct option. 

Three different recording devices were described by 
Pressey and his students. They developed a punch-board 
device, a system using chemically treated paper, and an 
electrical system. See Pressey (1950) for a more complete 
description of each. 

The type of device developed by Pressey suffers from at 
least two defects which seem quite important. First, the 
student must merely recognize the correct answer among 
a number of options. In many cases this format is quite 
unlike the real life situation for which the skills are being 
developed and the problem of transfer arises. Second, 
the materials which Pressey presented were not arranged 
in any hierarchic fashion. The lack of carefully prepared 
sequence of information does violence to one of the 
principles of learning described earlier in this paper. 

Recently, Skinner (1958b) developed a different type 
of teaching device’ which circumvents the limitations of 
the Pressey machines. Using the devices developed by 
Skinner, the learner must construct, rather than merely 
recognize correct answers to questions. Further, Skinner 
and his colleagues spent a considerable portion of their 


*Kendler (1958) has discussed some of the special problems 
of transfer connected with teaching devices. 
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research time, in attempting to construct programs of 
subject matter, beginning with simple problems and grad- 
ually increasing complexity only as the learner demon- 
strates that he is ready for it. For a good illustration 
of what is meant by a program, see Skinner (1958a, pp. 
3-7). 

One of Skinner’s machines is used to teach arithmetic. 
The learner constructs his answer by operating sliders, 
each of which represents a digit. In this way, the learner 
can construct answers up to four digits in length. For 
materials other than arithmetic, the learner writes out 
his answer. Then he may compare his answer with the 
correct one, merely by operating a lever. The reader can 
easily see that such a procedure introduces a source of 
error, since the learner decides whether or not his answer 
is correct. In some models, the machine provides for 
recycling of those questions which the learner had wrong 
on previous trials. 


PARAMETERS INFLUENCING THE EFFECTIVENESS 
OF AUTOMATED INSTRUCTION 


A good deal of the research on automated instruction 
has taken the form of evaluation studies, in which teach- 
ing by machine has been compared with teaching by the 
“usual” method. The present writer shares with Gilbert 
(1958, p. 29) the belief that while a certain amount of 
evaluation research is necessary in order to justify a 
continued interest in the basic concept of automated in- 
struction, the major portion of the research effort should 
be devoted to an experimental analysis of the parameters 
which influence the effectiveness of automated instruc- 
tion. This latter type of study will be,discussed first. 

A distinction may be drawn among three classes of 
variables of which the effectiveness of automated instruc- 
tion might bé a function. They are: characteristics of the 
device, of the program, and of the learner. Each of these 
classes of variables will be discussed in some detail and 
an effort will be made to summarize the pertinent re- 
Search findings. 
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Characteristics of the Device: The reader will recall 
that, earlier in this paper, a teaching device was defined 
as a machine by which stimulus materials are presented 
to a learner and which requires some response on his part 
in order for the device to operate. Further, the type of 
teaching device with which this paper is primarily con- 
cerned is one which provides immediate feedback to the 
learner. This type of analysis permits one to discuss the 
characteristics of the device within the framework of a 
three-fold analysis: 

1. Display or input characteristics: So many specific 
devices have been used by workers as means of presenting 
the stimulus materials that a complete classification of 
them is almost impossible. Suffice it to say that most 
workers have made use of mechanical, electrical, or optical 
components to present information to the learner. Two 
exceptions may be noted. Crowder (1958b) and Homme 
& Glaser (1958) have successfully presented stimulus 
materials in the form of a modified textbook. And Rath 
& Anderson (1958) have used the IBM “650” print-out. 

To the writer’s knowledge, no experiment has been 
performed which compares the efficiency of different 
methods of presenting stimulus materials. 

2. Response or output characteristics: Two basically 
different modes of responding have been employed. 
Pressey, and his colleagues, and some researchers in 
military establishments have made heavy use of the 
multiple-choice or recognition mode, while Skinner and 
others have used the “free operant” or construction mode 
of responding. No study was found which permits the 
comparison of these two basic modes of operation. 

However, Irion & Briggs (1957) have experimentally 
compared several variations of the multiple-choice mode 
of operation. The four modes compared were (a) quiz 
mode: in which the S chooses one of the responses by 
pressing a switch and the machine immediately indicates 
which of the responses was correct, (b) , modified-quiz 
mode: which is identical to (a) except that a buzzer 
indicates that an incorrect answer was made, (c) practice 
mode: in which the S must press switches until the correct 
response is made, and (d) single-try mode: in which the 
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S may respond only once and then the next stimulus item 
is presented. 

Each of the four modes of operation was employed by 
different groups of subjects as they learned three types of 
tasks: serial learning, paired-associate learning, and prob- 
lem solving. The dependent variables were the number 
of trials required to learn the tasks to a given criterion 
and a measure of retention, taken two weeks after learn- 
ing. 
th general, the relative efficiency of the four modes of 
operation was in the order: quiz-mode, modified-quiz 
mode, practice mode, and single-try mode. The amount 
of difference in efficiency was a function of the type of 
learning task, Later, Irion & Briggs used different modes 
of operation within a learning situation and found no dif- 
ferences in amount of time to criterion. 

3. Reinforcement characteristics: Little attention has 
been given to the problem of the optimal type and 
schedule of reinforcement to use as the means of main- 
taining the maximal rate of performance. Most workers 
report that sheer “knowledge of results” suffices to keep 
the learner working. Skinner (1958) reports that students 
work harder and longer when using the machine than 
they do when they merely read the textbook. The implica- 
tion is that “knowledge of results” is an effective rein- 
forcer, but it seems clear that various methods of pro- 
viding feedback might very well influence the learner's 
performance rate. For a more complete discussion of this 
subject, see Porter (1957, pp- 134-136). 


Characteristics of the Program: Programming tefers to 
the arrangement of the stimulus materials in the order 
of presentation which will maximize the rate of acquisi- 
tion of the habits to be learned. Presumably, proper 
programming permits the gradual shaping of the ee 
responses from low to high levels ofeapproximation 0 
the desired responses. t 

It should ie noted that Pressey and his group gave 
little attention to the problem of programming, perhaps 
because they were primarily interested in developing 
testing devices and only secondarily interested in develop- 
ing teaching devices. Further, it 1s likely that some types 


68 A Review OF THE LITERATURE 


of subject matter require little in the way of programming 
at all (e.g. trouble shooting and the identification of 
objects such as enemy aircraft). 

But for certain kinds of complex tasks, it is likely that 
efficient learning can take place only if the materials are 
carefully programmed into a hierarchy, extending from 
very simple to complex habits. See Gilbert (1958) for 
a more complete discussion of this point. 

Several problems associated with programming im- 
mediately present themselves. 

1. Extrinsic Versus Intrinsic Programming: Of those 
workers interested in teaching devices, Skinner was the 
first to give serious attention to the matter of program- 
ming. For a given subject matter, he attempted to con- 
struct a single program, through which all learners would 
proceed. This type might be thought of as a straight-line 
program. Later, Crowder (1958a, 1958b) described an- 
other type of programming which tailors the program to 
the individual learner, depending upon his strengths and 
weaknesses (as reflected by his previous errors), Crowder’s 
might be thought of as a branching program. The 
branches would permit the learner to retrace his steps 
back through that portion of the program which his 
errors indicate that he did not adequately learn, Crowder 
referred to his type of program as intrinsic and he re- 
served the term extrinsic program for the type described 
by Skinner. 

For obvious reasons, the intrinsic program would place 
a special burden on both the author of the program and 
on the teaching device used. However, the use of an 
intrinsic program might very well result in a more efficient 
learning situation. Unfortunately, no experiment has yet 
been done which compares the two methods of program- 
ming directly. 

2. Size of Step: Skinner (1958) and others feel that 
the most effective program is one which raises the level 
of the learner’s proficiency by such small steps that the 
probability of the leamer being wrong is sd low as to be 
negligible. In fact, Skinner uses the percentage correct 
as a measure of the optimum size of step. 

On the other hand, some have argued that the optimal 
size of step is the one which results in the most rapid 
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learning to a given criterion and that this optimum can 
only be determined experimentally. Homme & Glaser 
(1958) investigated the effect of size of step upon fre- 
quency of errors and upon immediate and delayed reten- 
tion, Using a single program on elementary number 
theory, they varied the number of steps over four values: 
30, 40, 51, and 67. Presumably, the greater the number 
of steps, the smaller the mean size of step. Independent 
groups of five Ss each were used. The results showed that 
the smaller the step, the fewer the errors and the greater 
the retention. However, it is conceivable that increasing 
the number (and therefore decreasing the size) of steps 
beyond 67 might have resulted in no further change in 
dependent variable performance. Further, the relation- 
ship between measures of learning and size of step might 
interact with characteristics of the program employed. 
Clearly, further research is necessary before this important 
question can be answered unequivocally, but a start has 
been made. 

3. Rules or Principles of Programming: Skinner 
(1958), Beck (1958), and Gilbert (1958) have de- 
scribed a number of rules or principles of programming. 
Although these rules or principles seem eminently logical, 
they appear to lack empirical support. Thus, they might 
better be thought of as hypotheses to be tested. Space 
does not permit a detailed discussion of these tules and 
the interested reader is referred to the. appropriate 
teferences. 


Characteristics of the Learner: A third major variable 
of which the effectiveness of automated instruction might 
be a function has to do with some characteristics of the 
learner, For example, one might hypothesize that the use 
of teaching devices might wipe out differences in achieve- 
ment measures associated with intelligence or aptitude 
test performance. The findings of a tiumber of experi- 
ments seem to support this hypothesis. 

Porter (1958) found that the correlation between IO 


and achievement in spelling was lower for a group of 


le ht by machine than for a control 
e ieee 1 fashion. Ferster & 


group who were taught in the usua 
Sapon (1958) Sta similar findings on the relations 
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between aptitude and achievement in a course in Ger- 
man. Finally, Lumsdaine (1958) showed that the ad- 
vantage of active participation in a simple learning task 
decreases with increases in intelligence. 

The experiments just cited might lead one to believe 
that superior students do not learn efficiently by machine. 
Pressey (1950), Briggs (1947) and Jensen (1949) have 
all reported data which show that bright students are 
quite capable of benefiting from the use of teaching 
devices. 

‘Another characteristic of the learner which might in- 
fluence the effectiveness of automated instruction is the 
level of anxiety of the learner with respect to the subject 
matter being taught. Porter (in a personal communica- 
tion) has suggested that learners who are quite anxious 
or who have a low level of aspiration concerning the 
subject matter seem to be ones who profit a great deal 
from machine instruction. It seems reasonable to suppose 
that since the machine instruction provides for many 
reinforcements, the learner’s degree of anxiety might be 
reduced and his level of aspiration raised. The result 
should be an increase in rate of learning, but this hypothe- 
sis has yet to be tested. 


THE EVALUATION OF AUTOMATED INSTRUCTION 


A good number of studies have been located, which 
can be used to compare the effectiveness of teaching by 
machine with teaching by the “usual” classroom tech- 
niques. A number of these studies were performed using 
one or another variation of Pressey’s multiple-choice type 
of device. All studies conclude that there is something 
to be gained by using machine type of instruction. Usu- 
ally, the advantage is expressed in terms of the superior 

erformance of an experimental group (those using teach- 
ing devices) over a control group. 

Porter (1957, pp. 137-40) has carefully reviewed most 
of these studies and has pointed to a number of defects. 
In many of the studies cited, little if any effort was made 
to equate the groups for initial proficiency with respect 
to the subject matter, for level of motivation, for the 
“novelty” effect, oF for the amount of time spent in 
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study. Further, although immediate reinforcer teaching 
devices were used in all of these studies, it cannot be said 
that these studies demonstrate the superiority of im- 
mediate versus delayed reinforcement since it seems clear 
that the control groups received less reinforcement as 
well as delayed reinforcement. 

Several evaluation studies have been performed using 
Skinner’s type of device in which the learner constructs 
his answer (rather than merely identifies it) and which 
presents the stimulus materials in a carefully arranged 
hierarchic sequence. Porter (1958) has shown that the 
use of such a device and program for the teaching of 
spelling in second and sixth grade students renders them 
superior in spelling achievement to a control group. 

Homme & Glaser (1958) recently reported a study 
in which comparisons were made between groups who 
learned by using programmed textbooks and groups who 
used standard textbooks. Two types of subject matter 
were used: sections of a statistics course and a course in 
the fundamentals of music reading. In both cases, the 
experimental groups outperformed the control groups on 
achievement tests. No mention is made in this study of 
the procedures used to equate groups for initial pro- 
ficiency, amount of time spent in study, and so on. 

Holland (1958) and Ferster & Sapon (1958) have 
also reported the results of automated instruction, using 
college students as subjects. Holland had students learn 
sections of a course in psychology and Ferster & Sapon 
used the subject matter of a course in German. Control 
group data were not reported. 

On the basis of this cursory review of the validity 
studies, one might conclude that we are still lacking in 
carefully executed experiments which unequivocally dem- 
onstrate the superiority of automated instruction over the 
“usual” classroom procedures. And teo, one might say 
that we know even less about why automated instruction 
might prove to be superior. It is entirely possible that all 
of the validity studies are dealing with a “novelty” effect. 
However, Porter (1958) has shown in his year long 
study of spelling achievement that the second semester’s 

erformance of the experimental group was just as 
superior to the control group as was the first semester’s 
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performance. Such evidence would suggest that if there 
is a “novelty” effect operating in Porter’s study, it is slow 
in disappearing. Nevertheless, it is very clear that a great 
deal of additional research is in order before we can: be at 
all confident about the superiority of automated instruc- 
tion over the “usual” classroom procedure. 


A FEW HYPOTHESES TO TEST 


The review of the literature presented in the previous 
section of this paper provides a number of hypotheses 
which seem worth testing. This discussion of hypotheses 
might well be organized around the three major classes 
of variables of which the effectiveness of automated 
instruction is a function. 


Characteristics of the Device: The writer shares the 
view expressed by Gilbert (1958a) that, for the present, 
most of the research on automated instruction might best 
be spent by studying relations between characteristics of 
the program and rate of learning or amount of retention. 
However, at least three characteristics of the teaching 
device might profitably be examined immediately. 
Hyporuesis 1. The use of self-paced instructional de- 
vices yields more rapid learning and better retention than 
does group-paced devices and that this relationship holds 
for all types of learning tasks and for all types of learners. 

Lumsdaine (1958) has suggested that group-paced de- 
vices might prove to be almost as effective as those which 
permit the individual leamer to proceed at his own pace. 
Although there is good rationale for the belief that in 
most cases the self-paced device will prove to be the 
more effective of the two, this is still an empirical 
question. Further, the advantage of the self-paced device 
over the group-paced type might interact with both the 
nature of the task to be learned and with a number of 
characteristics of the learner. Finally, the advantage of 
the self-paced device over the group-paced idevice might 
not be great enough to warrant the accompanying in- 
crease in original and maintenance costs. 

Hyroruesis 2. The use of reinforcers in addition to 
“knowledge of results” will result in a higher rate of 
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performance on teaching devices and perhaps render 
“knowledge of results” (as a reinforcement) more resist- 
ant to satiation. 


As mentioned earlier in this paper, most researchers 
seem to believe that mere “knowledge of results” is 
sufficient reinforcement to keep students working. How- 
ever, it is entirely possible that prolonged exposure to 
such reinforcement may result in an effect akin to stimu- 
lus satiation. Porter (1957) has warned of just such a 
“novelty” effect. 

Moreover, it may well be that the use of other rein- 
forcers in addition to “knowledge of results” might result 
in an increase in rate of performance. Skinner (1954, 
pp. 93-94) has suggested that approval from the teacher, 
being permitted to engage in desired activities, successful 
competition with other students, and even aversive stimu- 
lation might be used to augment the reinforcing properties 
of “knowledge of results.” 


Hyporuesis 3. There is an indirect relationship between 
the amount of delay of “knowledge of results’ and the 
rate of learning and this relationship holds for all types 
of subject matter and for all types of learner. 


Pressey (1950, p. 418), in particular, has argued that 
the superiority of teaching devices over the more standard 
pedagogic practices stems from the fact that the devices 
minimize the delay between the learner’s responses and 
“knowledge of results.” But Porter (1957, p. 138) has 
pointed out that in several of Pressey’s validating studies, 
the amount of “knowledge of results” varied simultane- 
ously with the delay in “knowledge of results.” Thus, we 
are left without knowing which of the two factors is the 
more important. Deese (1958, p. 199) reports a study by 
Saltzman which shows that a delay in “knowledge of 
results” can seriously hamper the learning of a rote 
verbal task. A delay of but six seconds resulted in a 50 
per cent increase in errors. 

However, it may well be that certain types of subject 
matter and perhaps certain types of learner might be 
able to withstand fairly long delays without affecting 
learning rate appreciably. Such information would be of 
some help to the designers of teaching devices. 
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Hyrortuesis 4. Having students learn to compose cor- 
rect answers to questions results in faster learning, better 
retention, and more positive transfer to “real life” situa- 
tions than does having students learn to Tecognize correct 
answers from among a number of options. Further, this 
relationship holds for all types of learning situations and 
for all types of learners. 

Skinner (1954) contends that, for most learning 
situations, it is more desirable to render learners capable 
of composing correct answers to questions, rather than 
make them able merely to recognize them. There are 
good reasons for believing that Skinner is correct on this 
point, but once again, this is an empirical question. Fur- 
ther, the advantage of composition over recognition may 


interact with the type of task and with characteristics 
of the learner. 


Characteristics of the Program: As mentioned earlier 
in this paper, the writer believes that the usefulness of 
teaching devices is more a function of characteristics of 
the program than a function of characteristics of the 
device, itself. For the Present, many believe that program 
writing is an art, which is perhaps another way of saying 
that we know little about the characteristics of the op- 
timal program. 

However, some Program characteristics have already 
received some study. Crowder (1958b) has distin- 
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to extrinsic ones, the intrinsic type places a considerable 
burden upon both the program writer and upon the 
teaching device, itself. One way of reducing the mechan- 
ical complexity of the device would be to use a pro- 
grammed textbook, intrinsically organized. Crowder 
(1958a, 1958b) has already put this idea to use. 


Hyrornests 6. The rate of learning via automated in- 
struction is a function of the size of step employed in the 
program. Further, if properly measured, the optimal size 
of step is a constant for all types of subject matter. 

When a student is taught a given subject matter, it 
is assumed that he starts off at a given level of proficiency 
and, after training, ends at a higher level. Now, one may 
attempt to go from the lower to the higher level by 
presenting the learner with relatively few or many steps. 
Further, assuming that the steps are roughly equidistant, 
then size of step is negatively correlated with the number 
of steps. Thus, both number of steps and size of step 
might be variables of which the rate of learning is a 
function. 

„Skinner (1958, p. 5) feels that steps should be so 
small that errors are rarely if ever committed. In fact, 
some workers feel that the percentage of errors is a meas- 
ure of the worth of an item. Items which result in more 
than a minimum number of errors are viewed with 
suspicion. ©n the other hand, some workers feel that 
some percentage of error might well be built into the 
program and presumably this would be done by increas- 
ing the size of step in order to obtain the desired per- 
centage of error. i, 

The question of the optimum size or number of steps 
in a given program is obviously an empirical one, to be 
answered by experiment. Homme & Glaser (1958) have 
shown that increasing the number of steps (and thereby 
decreasing their size) resulted in morc efficient learning. 
However, it is entirely possible that Homme & Glaser did 
not vary the number of steps in the program in an asymp- 
totic function or even in a decrease in learning efficiency. 
Clearly, this variable is worthy of careful examination. 

Finally, the writer believes that subject matter cur- 
tently thought of as being difficult may simply be ma- 
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terial which is programmed + in very large steps. If a 
technique could be developed which would permit the 
measurement of size of step in units which provide an 
equal-ratio scale, then one could test the hypothesis that 
the optimal size of step is a constant for all types of 
subject matter. Unfortunately, the writer knows of no 
such technique. 


Characteristics of the Learner: As mentioned earlier, 
several studies have shown that the amount of improve- 
ment shown by students is relatively uncorrelated with 
their intelligence and/or aptitude for the subject matter. 
Such data must be interpreted carefully. They do not 
imply that students with low intelligence or aptitude 
profit more from automated instruction than do “bright 
students. Rather, these findings merely state that the 
amount of profit is independent of such factors as intel- 
ligence and aptitude. 

Researchers would do well to continue to search for 
characteristics of the learner which are correlated with 
amount of improvement. For such information would 
probably give us hints about the optimal design of the 


program, the type and schedule of reinforcement, and 
of the teaching device, itself. 


Hyrotnesis 7. The rate of learning via automated in- 
struction is a function of level of anxiety and/or level 
of aspiration of the learner with respect to the subject 
matter. 

Porter® has noted clinically that one benefit of auto- 
mated instruction seems to stem from the fact that some 
learners become less anxious about their performance on 
the subject matter after they have used a teaching device 
for some time. Such an observation suggests that im- 
mediate knowledge of results and perhaps a high per- 
centage of correct Tesponses might be more beneficial to 
learners who are anxious than to those who are not. 


‘The reader should recognize that teachers end textbooks 
program material, in the sense that the material is arranged 
in a hierarchy conducive to rapid learning, 

® Personal communication. 
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NOTES ON DIFFICULTIES INHERENT IN RESEARCH 
ON AUTOMATED INSTRUCTION 


Porter (1957, pp. 137-38) and Gilbert (1958, pp. 
29-32) have discussed some of the difficulties inherent in 
research on automated instruction. Both workers feel 
that most evaluation studies presently available simply 
do not meet the necessary criteria of excellence which 
permit us to state categorically that automated instruc- 
tion results in performance superior to instruction via 
the “usual” techniques. Most studies simply lack suf- 
ficient experimental control. The experimental and con- 
trol groups differ in so many respects that it is impos- 
sible to tell which of the differences in treatment account 
for the differences in performance. 

The chief difficulty seems to stem from the inability to 
control the subjects” previous experience with, and apti- 
tude for, the subject matter to be learned. Most studies 
have used regular students as subjects in classrooms at 
all levels of education. Further, the subject matter em- 
ployed is usually standard material. Finally, many of the 
studies have employed such large numbers of ‘subjects 
that it is practically impossible to leam much about what 
is taking place while an individual is learning. > 

The writer suggests that at least some research time 
might be devoted to the careful observation of small 
numbers of subjects while they are learning materials with 
which they are totally unfamiliar. The classical non- 
sense syllable or objects frequently used in concept forma- 
tion studies recommend themselves. Zeaman (1958) re- 
cently discussed the use of the latter type of material 
within the context of automated instruction. s 

If, for one reason or another, the researcher desires to 
do research in a real classroom, he might give thought to 
the teaching of relatively discrete skilis which are sub- 
matter for only one or two years of schooling. For 
Je, at the elementary school level one might easily 

TO; at the teaching of “telling time” or the use of 
p oa numerals. Such instruction rarely interacts with 
one types of subject matter. At the collegiate level, the 
res aching of parliamentary rules might be more easily 
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programmed than the teaching of algebra, philosophy, 
and the like. 

Further, ease of programming is not the only benefit 
which might obtain from the use of subject matter de- 
scribed above. Since such material is relatively discrete, 
the techniques used to teach it will not interact with 
the learning of other subject matter. 
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Teaching Machine Dichotomy: 
Skinner vs. Pressey* 


Epwarp B. Fry 
Loyola University, Los Angeles 


Even though the field of teaching machines is not very 
old or very large, there has already developed what ap- 
pears to be a relatively major dichotomy. On one hand 
we have the historically older faction headed by Sidney 
Pressey at Ohio State University, which emphasizes 
multiple-choice devices; and on the other, a newer fac- 
tion headed by B. F. Skinner at Harvard University, 
which emphasizes construction of response devices. Nu- 
merous theoretical and practical considerations (amount 
of error permitted and methods of programming) enter 
into the dichotomy. i 

The common characteristics of teaching machines may 
be described simply. These are devices which tutor the 
student without assistance from a human instructor. They 
require the student’s active participation. He must re- 
spond to questions which may be interspersed with 
graphic or verbal subject matter, or with other techniques 
as the Socratic method or “vanishing : (gradually remov- 
ing part of a word or phrase until the student responds 


n= o 
ted at the annual convention of the American 
* i sas Pi Association, Cincinnati, September 1959. This 
Eoi olob Finally appeared in Psychological Reports, 1960, 6, 
artin z snd is used here with the permission of the author and 
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without a cue). Two basic principles are involved: (a) 
complete variability in rate, rate being determined by the 
student’s responses which permit individual differences, 
and (b) immediate knowledge of results, i.e., the device 
“rewards” the student with the answer immediately after 
he responds. 


TYPES OF QUESTIONS AND LEARNING THEORY 


The first major difference between the two schools is 
the type of questions preferred. Pressey prefers the mul- 
tiple-choice type of question, partly because originally 
he envisioned his teaching machine as also an automatic 
testing device. He states (1926) that permitting the 
student to make more than one error per question (his 
“degree of ignorance” per question) greatly increases the 
spread and significance of the total score. Learning 
theory at that time was strongly influenced by association- 
ism. Pressey referred to the laws of recency, frequency, 
and effect, but was also willing to use questions which 
called for “insight.” 

Skinner prefers that the student construct his answers 
rather than relying upon recognition. On his arithmetic 
machine (1954) the student responds by moving digit- 
marked sliders to indicate the correct answer. The con- 
structed-response type of question is in line with Skinner’s 
emphasis on the Principle of operant conditioning in 
which characteristic learning is controlled by rewarding 
the student after he has made the correct response. 
Skinner attempts to structure the situation so that re- 
sponses lead step by step toward the desired goal, thus 
shaping behavior through successive approximations: 


AMOUNT OF ERROR DESIRABLE 


Skinner would avoid the structuring of wrong answers 
in programming questions for his teaching machine. 
“Why cause any learning at all between the wrong answer 
and the right question?” he asks. He would be happy if 
every student got every question correct. 

Pressey, on the other hand, is willing to allow some 
error. Although specific amounts of error are seldom 


a 
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mentioned, Pressey would say (and most psychologists 
would agree) that the number of right responses should 
greatly exceed the number of wrong. 


SUPPLEMENT OR REPLACE OTHER 
TEACHING METHODS? 


Although perhaps a debatable point, the dichotomy 
extends to the instructional uses planned for the two 
types of machines, Pressey’s devices having been used. 
more as supplemental adjuncts to courses and textbooks, 
and Skinner’s as replacement for texts and classroom in- 
struction. For example, Porter (1958) used a simple 
write-in machine to supplant spelling instruction; “No 
spoken instruction was given.” By contrast, Pressey 
(1950) used multiple-choice questions with self-scoring 
punchboards to supplement regular instruction in éduca- 
tional psychology. [Incidentally, results of both studies 
favored the experimental groups. Neither Pressey nor 
Skinner would eliminate teachers, both being interested 
in making more efficient use of the teacher’s time in 
class.] 


IS INTELLIGENCE STRICTLY QUANTITATIVE? 


To Skinner, the difference between a bright and a dull 
student is largely one of quantity of learning. If teaching 
steps are small and understandable, a poor student can 
learn the same thing as a bright student; he just has to 
work longer. And when students find concepts difficult, 
Skinner believes (1958), the instructor has not presented 
the steps in small enough or correctly ordered units. In 
planning curriculum, the programmer must discover and 
properly order the units of information, or steps. | 3 

While Pressey does not deny the truth of Skinner's 
e concepts of inaner and venues his 

i are structured as though “superior” students 
pee different (Briggs, 1949; Jensen, 1949). 
Crowder (1958) actually presents more and different 

ne for poorer students. Pressey places no stress on the 
Stepa ness of the learning steps or the ordering of presen- 


tation. 


quantitativ 
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PRACTICAL APPLICATIONS 


While both sides of the dichotomy envision wide use 
for their devices, actual hardware and experimental studies 
are rather scarce. Pressey showed an actual multiple- 
choice testing and teaching machine at the 1924 Ameri- 
can Psychological Association meeting. And, from time 
to time variations have appeared, such as Peterson’s 
chemo-card in 1931 and Angell’s punchboard study 
(1949) in which 81 matched pairs of chemistry students 
used punchboards on regular quizzes. Angell and Troyer 
later developed a commercial punchboard which Science 
Research Associates published for a time. Pressey’s 1950 
study of over 1000 students using punchboards led to 
the production of the punchboard for use in the Navy 
(Hill, 1955). 

Besnard, Briggs, and Walker (1955) ‘developed for the 
Air Force several large multiple-choice devices for trouble- 
shooting. Electronics trouble-shooting Tesearch has pro- 
duced numerous other multiple-choice devices, such as 
the Tab Test (Glaser, 1954), Trainer Tester, and Auto- 
mat. Also, Skinner has a small battery of disc-type write-in 
machines which were used by Holland (1958). The 
Navy Special Devices Center has recently constructed a 
small battery of similar machines. Skinner has a slider- 
type of arithmetic-spelling machine produced by IBM 
Also at Harvard, Porter has several of the very simple 
write-in machines mentioned earlier, P 


DICHOTOMY IN NON-MECHANICAL DEVICES 


A new development is not to use machine: 

to use books and paper devices, Although aie all, but 
of the control of machines, these devi 

bility of rate, unit presentation 
edge of results. Yet non-mechanical deyi 
basic dichotomy discussed here. A; p 
multiple-choice paper device is Crowder’s ;° 

bled book, in which a bit of information b an, cam- 
a multiplechoice question. The student anes R 
question by turning to the appropriate Page; the a G 
page tells whether the choice was correct, and why PE 
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answer was wrong, further instruction is given, followed 
by a new question. If correct, the next problem is pre- 
sented. 

Ferster and Sapon (1958) describe a cardboard device 
which requires free response. A cardboard sleeve contains 
a slot which exposes one line of a question on a printed 
paper slip. The student writes his answer on a scrap of 
Paper, moves the cardboard down one line to expose the 
answer, scores himself, and proceeds to the next question. 
Another device is that developed by Homme, Glaser, and 
Evans (1958), a “programmed textbook” to which the 
student makes numerous subjective responses. He is given 
the answer on the next page, thus providing him with 
knowledge of results. 


SUMMARY 


We have seen that in the field of teaching machines 
there is a dichotomy of opinion with respect to response 
mode, step size, amount of error desirable, leaming 
theory, views on intelligence, and the supplementing of 
traditional instructional methods. What we need now are 
a lot of good experimental studies, both theoretical and 
applied in nature 
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EDITOR’S NOTE 


Three approaches to programming have 
been developed; because of limitations of 
space only that used most frequently is repre- 
sented in this collection of readings. In David 
Klaus’ article which follows only linear pro- 
gramming utilizing a constructed-response 
format is discussed in detail. This has been 
called the Skinnerian type of program. The 
second approach is called “intrinsic program- 
ming” or branching and it usually is identi- 
fied with Norman Crowder (1959). In this 
technique, the student is given a unit of 
information containing a question or a prob- 
lem followed by a set of multiple-choice an- 
swers from which he selects one. If his choice 
is correct, the student continues on to the 
next unit of information, if his choice is in- 
correct, he is instructed to go to a sub-pro- 
gram (in a scrambled book, to a page) which 
explains his error to him and instructs him 
to try the same or a similar problem again. 
The third approach combines the linear or 
sequential technique and the branching tech- 
nique by instructing the student to skip a set 
number of frames if his solution to a prob- 
lem is correct. The difference between the 
Crowder and the Skinner approaches is not 
mere mechanics—It involves a distinct dif- 
ference in philosophy of teaching. A thor- 
ough discussion of these approaches to pro- 
gramming may be found in Rigney and 
Fry (1961). 


Part Il 


& J 
nm ey 

x 

=e Calcutta > K 

1B. = RO 

7 = 


The Art of Auto-Instructional 
Programming 


Davm J. Kraus* 
American Institute for Research 


With the possible exception of large-scale group test- 
ing, no one development in the field of psychology seems 
to have as much potential for the better utilization of 
human resources as auto-instructional methods. The tech- 
nique has promise for producing a genuine and large scale 
improvement in educational practices and, as a conse- 
quence, it is expected to have a tremendous impact on the 
quality of classroom instruction. Because of the novelty of 
the technique and its potential impact on ‘education, it 
is not at all’ surprising that auto-instruction is more 
frequently talked about than understood, and that more 
predictions are made as to its potentials than facts are 
collected as to its capabilities. 

Perhaps more than anyone else, the professional edu- 
cator should keep himself well informed as to where 
auto-instruction has been and where it is going. He 
should know not only the facts and «the forecasts, but 
since he has the responsibility for wisely using these 


* This article is based on a paper read at the Seminar on 
Educational Research at Cornell University in December 1960. 
This paper originally appeared in Audio Visual Communica- 
tion Review, 1961, 9, 130-142, and it is reprinted here with the 
permission of the author. 
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materials, it is especially important that he become ac- 
quainted with the art of programming—the preparation 
of auto-instructional programs. . 

Sooner or later, educators and teachers are going to 
have full responsibility for writing programmed materials 
for classroom use, but unfortunately, programming is 
still a skill about which we know very little. The rules we 
have are incomplete at best, and for this reason, perhaps 
the greatest need at this moment is for a clarification of 
the techniques involved in preparing auto-instructional 
materials. Efforts along these lines will not only facilitate 
increased research and development, but will permit the 
preparation and rapid utilization of a large number of 
programs in our schools. But before illustrating what 
appear to be important characteristics of good as opposed 
to poor programs, it may be helpful to describe a little 
about what programming is, where it has been, and where 
it may be headed. 

In essence, the auto-instructional method promotes the 
orderly and controlled development of an individual’s 
skill in much the same way as a good tutor might do. 
By presenting lessons in small, carefully sequenced steps, 
complicated skills can be developed by gradually pro- 
gressing from very simple to very complex levels of pet- 
formance. Since the student must perform actively at 
each step during training, it is possible to carefully guide 
the development of his skill by means of ‘immediate con- 
firmation as. to the correctness of each response. Further- 
more, the auto-instructional technique facilitates the 
evaluation and improvement of the materials during the 
course of their development in that the difficulty level 
or contribution of each step can be carefully ascertained 
and, when necessary, any step may be modified ot 
Tevised. 

In Fig. 1 three frames from physics are used to illus- 
trate how programmed materials of the kind we have 

cen preparing at the American Institute for Research 
appear to the student. As illustrated in Frame 1, the 
student is exposed to material in the form of a small 


step which is designed in such a way as to encourage 
him to respond. Before beginning the program, the stu- 
dent is told 


to respond while he reads through the frame. 
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In general, he is to write his answers either directly in 
the book where the frame appears, on a separate answer 
pad, or if an auto-instructional device or “teaching ma- 
chine” is used, on a strip of paper. As soon as he has 
completed the three responses called for, the student can 
expose the correct answers either by turning a page or 
advancing the machine. There, the student would see 
the three answers, heat, energy, and heat, which he can 
check against the responses he had given. Then, the 


A fire can keep us warm because it gives off energy 
Frami in the form of ... The sun also 

gives off ..... «+ in the form 

OF o:a:s:0ie ere sisiwinieieeeieiece 

Heat is one form of .. ee eecccecevescrces 
Frame Physicists call it thermal energy. A quart of boiling 

water has more .....eeeeeeeeeecserees Energy 

than a quart of cold water. 

As a warm object grows cooler, it loses heat, that is, 

y it loses «0.2.2.0. seeeeeeeeeees energy. When a 

Frame’ 3 A i 

cool object gets warmer, it gains 

that fs, it. galns™ sccsecvscaccce 


*The asterisk indicates that more than one word belongs in that 
space. 


Fig. 1. Three frames from a program in physics. 


student would again turn the page or advance the pro- 
gram and look at the second frame, shown in Frame 2. 

The student again would respond and then check to see 
if his answers are correct. He would then turn to the 
third frame, shown in Frame 3. Learning that heat is 
energy, or more specifically, thermal energy, is not very 
astonishing in itself. But in terms of a high school 
physics prog:am of 15,000 or 16,000 frames, it can be 
seen how each small step gradually and systematically pro- 
duces 2 substantial amount of learning. 

Research already accomplished indicates in a general 
way what can be achieved by means of auto-instructional 
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programs. Under the sponsorship of the Office of Edu- 
cation, Dr. Lumsdaine and I prepared six weeks of high 
school physics in auto-instructional form. From these 
studies we were able to conclude that when used as a 
supplement to an. otherwise full program of instruction, 
Programmed materials do add significantly to student 
achievement. Furthermore, the evidence we obtained also 
suggested that auto-instructional programs may be relied 
upon to provide instruction, in physics at least, inde- 
pendent of classroom lectures and recitation. 

The use of auto-instructional programs has not been 
limited, of course, to developing proficiency with the facts 
and concepts of physics. Hoiland and Skinner at Harvard 
University have Prepared a considerable portion of the 
introductory psychology course in programmed form. 
Glaser and his associates at Teaching Machines, Incor- 
porated have prepared a program to teach descriptive and 
inferential statistics at the college level, and Eigen and 
Komoski! at Collegiate School have prepared a program 
to teach logic to children at the elementary school level. 

Industrial organizations have also begun to explore the 
value of auto-instructional materials for their technical 
training programs. We recently assisted a group of psy- 
chologists and instructors at IBM as they began the prep- 
aration of programs to teach the operation and mainte- 
nance of equipment as well as the programming of giant 
computers. At HumRRO in Washington, Rocklyn has 
Prepared a Program which provides instruction in oral 
Russian which can be used, despite the limited vocabulary 
Involved, to deal with Russian speaking individuals; 
Shettel and Angell at the American Institute for Research 

ave Prepared a program to teach certain perceptual 
skills involved in the Operation of a SAGE center to Air 
Force trainees; and Audrey Holland has prepared a 
Program at the University of Pittsburgh to provide pre- 
liminary corrective ear training to students with articula- 
tory speech defects, 

€ now have reason to believe that even, higher order 
skills than these can be taught by means of auto-instruc- 


* Mr. Komoski has been named president of the newly estab- 


lished Center for Programed Instruction, 365 West End 
Avenue, New York, N.Y. 
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tion. We are currently engaged in an investigation of 
methods for teaching art judgment. While the program 
being prepared will include some information on art 
history and principles, it has as its major purpose the 
development of a feeling of appreciation for art and the 
ability to differentiate good art from that which is not. 
We are also attempting to develop a program to provide 
laboratory instruction in electronics which will involve the 
use of actual laboratory equipment and a program to instill 
in students the insight necessary to solve geometric 
theorems which they have never before seen. 

The success we have achieved thus far in broadening 
the applications of auto-instructional programs has led 
many of us to a feeling of cautious enthusiasm with 
respect to the progress that has been made. As more 
research is completed and more programs developed, we 
should have a still better understanding of the role that 
auto-instructional methods might have in the classroom 
of the future. The research that will contribute toward 
this goal must be carefully done. One of the most encour- 
aging characteristics of auto-instructional materials is 
that they almost uniformly are capable of providing in- 
dividual instruction, no matter how well or how poorly 
the program is written. All programs are not, and most 
certainly will not be, of equal quality. Until more ob- 
jective and empirical measures of auto-instructional effec- 
tiveness become available, the educator must be ade- 
quately sophisticated in his judgment to evaluate the 
teaching competence of a program he might use in the 
classroom. eats 

The educator must also be aware of the pitfalls in doing 
research on auto-instructional technology, both with regard 
to his own work and to evaluating experiments by others. 
The results of a study which depends on a poorly written 
program should be interpreted accordingly. Programming 
is at least as much of an art, at the current time, as clinical 
psychology, for example. No matter how fine an experi- 
menter one is, he is not in a position to evaluate clini- 
cal patients ‘using a Rorschach or base a study on his 
interpretation of projective test protocols unless he has 
been thoroughly trained in clinical techniques. Similarly, 
despite how well the experiment has been designed, its 
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implications for a science of programming are limited if 
the program writers have not had sufficient experience. 
This is especially true when the sequence of frames is 
very small. A surprising number of studies have been 
carried out using a “program” only 50 or 60 frames long. 
The validity of the conclusions drawn from such experi- 
ments should be strongly questioned. 


PROGRAMMING STEPS 


Now to return to the problem of how programs are 
written. Because it is the area with which I am most 


familiar, I will describe the procedures we have been . 


using to program physics at the American Institute for 
Research. It should be noted, though, that the technique 
for preparing programs varies from individual to indi- 
vidual and, furthermore, that it is frequently modified 
because of the nature of the subject matter involved. 
Our first step is to list with care the specific objectives 
we will attempt to achieve with the program, expressed, 
in clear, behavioral terms. Most often, these objectives 
represent criterion behaviors, that is, the particular re- 
Sponses we would like to see in the student’s repertoire 
when he completes the program. For academic material, 
criterion behaviors often can be expressed in terms of a 
comprehensive test Covering the entire course. 
It is at this point that we make some initial decisions 
as to what kind of program will be constructed. For 
„ example, if the test (and the goal which it represents) 
requires only the ability to recognize the correct peste 
tive as on a multiple-choice test, the programmer wil 
attempt to achieve only this goal. It is not necessary, then, 
for him to prepare frames requiring constructed responses 
similar to those in the preceding figures. In the same 
way, if the student is required to know how to use a 
Particular formulë but not remember it, the programmer 
will plan to teach him just that, and no more. It has 
een Our experience that, for academic subjects at least, 
the educational goal usually is to provide the student with 
the skills necessary to Tecognize a given principle when 
he comes across it, express the principle using his own. 
words, and apply the Principle to a new situation Or 
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problem. Typical criterion behaviors which might be 
listed for a unit on heat, for example, might include such 
questions as: What is heat? How is it measured? What 
is the principle of linear expansion? Show how Boyle’s 
law might be applied, and so forth. 

Once the objectives have been identified, the next 
step is to carefully prepare a course outline covering the 
material that we wish to teach. The outline is frequently 
derived from a number of textbooks, reference sources, 
and from the comments of teachers familiar with the 
subject matter area. In addition, we have found that it 
has been highly desirable for us to obtain the assistance 
of a technical expert in the subject matter field. This 
expert is given the task of reviewing the materials pre- 
pared at each phase of the program’s development and 
is indispensable in preventing the dissemination of errors. 
We have also found that a well-qualified expert fre- 
quently can supply a large number of specific examples 
and interesting illustrations which can be incorporated 
into the program. 

The third step in our procedure is to begin the prepara- 
tion of draft frames. One of our programmers is assigned 
to each major unit and he or she is responsible for 
drafting groups of frames which cover the subject matter 
contained in the outline in such a way as to achieve the 
behavioral objectives of the course. We find that the 
process of drafting frames often points to inadequacies 
in the outline. Breaking down material into very small 
steps frequently identifies not only omissions in the out- 
line but instances where it contains material which does 
not reflect the course objectives. 

The draft frames are then edited three times. The 
first editing is done by another programmer who attempts 
to simplify the program and discover errors which the 
original programmer has made. The second editing is 
done by the technical expert who reviews the frames to 
insure their technical accuracy. A third editing is done 
y someone skilled in writing. Frames, just like text- 
books, can either be interesting or dull. Our third editor 
Teviews frames, rewording those that are awkward or 
unclear and revising others so that the program is enjoy- 
able to work with as well as instructional. 
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At this point, the most important step in the prepara- 
tion of a program is begun. The frames are shown to a 
trial subject who proceeds through the program as a 
student might eventually do in a classroom. Each answer 
is carefully recorded and, on the basis of the data ob- 
tained, the program is revised as much as necessary by 
either adding new frames or tewording those already in 
the program. It is at this step, possibly, that the difference 
between auto-instructional techniques and what many 
teachers do in the classroom is most pronounced. In 
programming, we assume that the burden of instruction 
tests with the program and not with the learner. Thus, 
when the student makes an error, it is assumed to be the 
fault of the program and not some inability on the part 
of the trial subject. ; 

When the frames are revised, the trial procedure is 
Tepeated. In fact, 10 or 15 trial subjects are often used 
in the preparation of a single sequence of frames. The 
programmer is rarely satisfed that his materials are as 
good as they can be until trial students not only show 
that they have mastered the material in terms of how 
well they do on the criterion tests, but can proceed 
through the program without making an error. Follow- 
ing the last revision of the program, the frames are again 
teviewed by the technical expert and only then are they 
teproduced for field tryout. 

The most difficult step in this procedure is the actual 
writing of the frames. As I noted earlier, we have barely 
begun to understand what makes a good frame. We do 
have a number of leads, however, that can be derived 
from laboratory studies of animal learning. I have dis- 
covered that educators often have a particular aversion 
to discussing the problems of classroom education in 
terms of how animals learn in the laboratory, but watch- 
ing a Tat, pigeon, or a chimpanzee learn can be a very 
Instructive experierce. The laboratory animal is not bur- 
dened with problems of low aptitude, inadequate readi- 
ness, lack of interest, or an unfortunate home life. In- 
stead, the experimenter accepts the animal as is and 
modifies his own behavior to the extent necessary tO 
produce learning. 


€ aspect of animal learning with particular impor- 
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tance for preparation of an auto-instructional program 
is the methods used for initially generating the correct 
response. Not only are these methods important with 
respect to auto-instruction, but perhaps one of the prime 
objectives of research in education today is the discovery 
of techniques for getting appropriate responses to occur 
initially. Probably this is our most ineffective skill in 
education and training. In teaching swimming, for in- 
stance, it is often quite difficult to get the correct response 
to occur the first time. The same is perhaps equally true 
of physics or spelling. The fewer similar or equivalent 
responses the student has, the less likely the responses are 
to occur at the appropriate time in the training schedule 
for practicing and reinforcement. 

In the laboratory, we often wait until the animal 
makes a response before reinforcing him. We have 
found, though, that it is not necessary for the animal 
to make a complete response, but only make responses 
which are preparatory or similar to the desired response. 
If we reward these preparatory responses, a procedure 
called successive approximations, we speed training since 
the animal is taught behaviors which make the criterion 
behavior more likely. ` 

Another procedure is to control the environment in 
such a way as to prevent erroneous responses from 
occurring. For instance, we can reduce the size of the 
apparatus until almost the only response that the animal 
can make is the desired one. The third possibility is to 
build on the animal’s previous learning. We can teach 
a rat to run through a complicated maze quite readily if 
we first train the animal to follow lights and then place the 
lights appropriately in the proper alleys of the maze. By 
and large, these procedures are not systematically em- 
ployed in education. For example, while we try to limit 
the environment in the laboratory, teachers are often 
encouraged to decorate their classroom: and provide other 
stimulation which, in fact, produce unwanted responses 
tather than gliminating them. I have always been struck 
by the amount of time a teacher spends in carefully dec- 
orating her room and the subsequent difficulty she has 
in maintaining the attention of students who are at- 
tracted to the displays she has put up. 
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Despite the rather obvious implications these labora- 
tory studies have for improving instruction, a surprising 
number of programs have been constructed on the philos- 
ophy that the student is bound to make errors while 
learning. These programs typically incorporate a feature 
which is called branching, that is, alternate sequences 
to be used by the student after he has made an error. 
The same philosophy leads to the construction of devices 
which are capable of preventing cheating. One wonders, 
though, whether branching or cheating must be a problem 
in education. After all, since we want the student to 
practice being right, why not help him as much as we 
can? In preparing auto-instructional materials, then, the 
programmer generally makes every effort to carefully 
guide the student through the required learning. He de- 
signs the program so that it plays an active role in 
insuring that learning will take place and that the proper 
responses will be those that are practiced. 

This particular approach to learning, which depends 
largely on placing the pupil in a carefully controlled 
environment, is not necessarily alien to education. Credit 
for the suggestion that instruction might be automated 
by properly arranging the circumstances in the student’s 
environment is often given to Socrates or Pressey or 
Skinner. It is my own feeling, though, that many of our 
current notions on auto-instruction were accurately antici- 
pated by John Dewey. To him, learning came about by 
doing; not random doing, however, but doing under 
such careful control that the student would necessarily 
meet with success when he was exposed to the conse- 
quences of his actions. Controlled or guided practice 
which leads to reward is the ‘central theme in both 
Dewey’s approach and that used in auto-instruction. 


PROGRAMMING RULES 


Now, let us look at some of the rules we have for 
building frames. While these rules generzlly represent 
the theory of auto-instructional programming rather than 
experimental evidence, they illustrate the auto-instruc- 
tional method and thus might be a help at this time. The 
first is, that the student learns from making a response 
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and not from hearing or seeing it. Only by practicing 
a response will a student learn and retain it. 

In programming, the question often arises as to whether 
it is important that the student make his response overtly 
or whether it is sufficient for him to make a covert 
response. With respect to simple verbal material, there 
appears to be no difference produced-by these methods. 
On the other hand, if the goal of the program is to 
develop some manual or perceptual skill, active respond- 
ing seems to be very important. The same is true when 
the student is apt to over estimate the ease with which 
he learns. Only when the student writes out a new and 
unfamiliar word does he receive the necessary practice in 
spelling it. Pronouncing the word to himself will not 
likely yield as satisfactory a result. 

Another question that often arises is whether to re- 
quire multiple-choice or constructed responses. Since they 
are capable of doing so, programs can and should encour- 
age the student to express his ideas in his own words. We 
then are sure that the student’s practice is of the kind 
we usually want to generate; by means of constructed 
responses we require the student to use and explain what 
he knows so that he practices presenting his ideas and 
his knowledge instead of simply recognizing a correct 
answer from among the alternatives given. There is still 
one more advantage to have overt, recorded responses. 
This is to provide the programmer with the data neces- 
sary for further revision of his program. The first rule of 
programming: require active responding. ‘ 

The next concern of the programmer 1s how to insure 
that the student will make a correct response, especially 
the first time that particular response is required. Aside 
from the laboratory methods described earlier, the pro- 
grammer can get assistance in this matter by studying 
the techniques used in advertising. In most instances, 
the advertiser is primarily concerned with producing the 
initial response. Some of the more effective advertising 
we have is based on the contiguity learning principles of 
Guthrie and Estes. Their theories suggest that a given 
Tesponse can be readily produced providing we mass 
sufficient cues and present them all at the same time. 
When driving down the highway, a motorist is apt to 
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come to an intersection where his stopping is made 
rather certain due to the presence of a stop sign, a 
flashing red light, and appropriate yellow lettering on 
the pavement. 

In programming, each frame must contain sufficient 
cues to produce the desired response. Sometimes we 
can get the proper response to occur by the use of 
wording; at other times, we may use thyming, synonyms, 
or particular word patterns we can assume are already 
in the repertoire of the students involved. j 

Appropriate cueing is one of the essential charac- 
teristics of a good frame. In fact, this is the essential 
difference between a test question and an auto-instruc- 
tional frame. In giving a test, we design the items in it 
to at least partially lead the student astray. We don’t 
want every student to get every answer correct. In a well- 
constructed program, however, every effort is made to 
insure that every student gets every response correct. 
The second rule: proper cueing. 

The fact that a student can make a correct response 
is rarely of interest to us unless the response is made in 
the appropriate context. Little Johnny may be doing an 
excellent job of pronouncing “daddy,” but after the first 
few times we hear it, we fail to think much of this 
response unless it occurs in the presence of the proper 
man. As learning proceeds during the course of using an 
auto-instructional program, the student must be gradu- 
ally weaned from the liberal cues we have supplied, so 
that he can make the correct response without our help 
in the presence of the relevant context. Third rule for 
writing a good frame: appropriate context. 

Together, these first three rules can be combined to 
describe a good frame. Basically, a good frame consists of 
only four parts. First is the response, selected beforehand. 
This is the part:of the frame that must be left blank. 
Too frequently, a beginning programmer finds that even 
though he has selected a satisfactory response, he con- 
structs the frame in such a way that sdine irrelevant 
word rather than the desired response is practiced. The 
second portion of a good frame contains whatever cues 
are necessary to reliably produce the desired response. 
Neither overcueing nor undercueing is desirable. The 


Davm J. Kraus 101 


third characteristic of a good frame is that it contains 
some relevant context, that is, the kind of stimuli we 
hope will be capable of evoking the desired response at 
some time in the future. Early in the program, context 
will probably be at a minimum and cueing will be quite 
prevalent. By the end of the program, however, cueing 
should have vanished so that the appropriate responses are 
made solely in the presence of the appropriate context. 
The fourth ingredient of a frame is optional. We call it 
enrichment. We have found that, in many instances 
interest in a program is heightened by inserting facts such 
as names or dates which are relevant to the material being 
covered but not part of the course objectives. Aside from 
these four ingredients, everything else in a frame is 
superfluous. 

The next characteristic of a good program is that it 
proceeds in very small steps. This is a rather difficult 
point to get across to a new programmer. Several studies 
have indicated, however, that the more steps, the better. 
In fact, the data seemed to indicate that a student 
proceeds faster through a program purposely made longer 
by the division of frames into smaller units than he does 
through a very condensed program. The number of 
frames in a program is not necessarily a good index of 
its efficiency, but as is the case in many other areas, 
one seems to get what one pays for, that is, the more 
frames the better. Rule four: small steps. k 

As the program is developed, the role of careful se- 
quencing becomes evident. Most subject matter consists 
of material of unequal difficulty. Complex concepts are 
built on simple concepts. Long involved formulae often 
can be shown to represent several shorter formulae com- 
bined together. In preparing a program, the writer can 
greatly enhance its efficiency by thoroughly determining 
beforehand what the sequence of topics will be. The 
fifth rule of programming: careful sequencing. 

Another problem having to do with the program as a 
whole is the problem of how often and how frequently 
a response should be practiced. As was the case with the 
number of steps, the tule here seems to be that the greater 
the number of repetitions, the better. The repeated review 
of a given response is almost universally more irritating 
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to the programmer than it is to the learner. In the first 
place, the program writer already knows the response. To 
him, frequent repetitions are terribly tedious. From the 
student’s point of view, however, repetitions can be ter- 
ribly important. It is one of the basic ptirciples of 
learning that retention of a response depends quite heavily 
on the amount of overlearning that has taken place. Thus, 
when a student complains of having made a certain 
response too often, we should be careful not to delete so 
much practice with that response as to affect its retention. 

One guide to assure that repetition is frequent enough 
is that the programmer, in reviewing his own frames, 
should be sensitive to his echoic behavior. When a par- 
ticular response stops ringing in his ears, it is probably 
time to review that response in the program. It is prob- 
ably helpful, in addition, to introduce as much variation 
as possible in the cueing and context associated with a 
particular response each time it is repeated. This kind 
of variability can be expected to reduce monotony and 
also to induce a desirable amount of generalization with 
respect to that response. The sixth rule for programming: 
frequent repetition. 

The next rule concerns the technical accuracy of the 
material contained within the program. The preparation 
of an auto-instructional program is a great amount of 
work. Too often, a writer engages in the necessary effort 
but fails to realize that the very excellence of his program 
in a psychological sense insures that his students will 
learn well, but learn the wrong facts and concepts. In- 
suring technical accuracy, of course, also involves the 
inclusion and exclusion of material. It is often the case 
that a programmer attempts to include too much into 
his program to permit the proper coverage of any one 
topic. In general, it is probably more advisable to do a 
thorough job on somewhat less material than an incom- 
plete job on a larger amount. The seventh rule: knowl- 
edge of the subject matter. 

Now we come to a few don'ts. Probably the most 
common mistake made by a beginning programmer is 
that he prepares the frames as if he were a lecturer 
presenting information. 

The lecturer can be contrasted with the tutor or auto- 
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instructional program in that he guides the student in 
what to learn but then delegates to the student the 
responsibility for structuring the material in small steps, 
for sequencing these steps, for scheduling sufficient 
practice, and for meeting with the success that will serve 
to reward desirable responses. Imitating a lecturer, the 
inexperienced programmer writes a paragraph for the 
student to read and follows it with a question which the 
student is expected to answer. Frame A shown in Fig. 
2 is an example of this type. 


“O E E 
Temperature and heat have different meanings. 
Heat, the amount of thermal energy stored in an 
object, depends on the mass of the object and what 
Frame A it is made of as well as its temperature. Heat is 
measured in calories, but since heat and temperatyre 
are not the same, we do not measure . 
in calories. 


In general, liquids, solids, and gases will expand 
Frame B when heated. Kerosene is a ...eeeeeeeeereveees 
it will expand when heated. 


Most metals contract when they solidify, but there 
are exceptions to this rule. For example, bismuth 
and antimony do ...eseeeeeeeeeeeeees contract 
when they go from a liquid to a solid state. 


Frame C 


Copper, iron, wax, and glass all contract when they 
Frame D solidify. Most substances, in fact, ....+seeseceeece 
when they go from a liquid to a solid state. 


An astronomer, Anders Celsius, invented the centi- 

grade scale of temperature. A thermometer which 
Frame E reads 100 degrees in boiling water is measuring 

the temperature of the water according to the 
. scale of temperature. 


The centigrade scale of temperature is more prop- 
Frame F erly called the ....ssseeeeeeeeess scale, after 
the name of its inventor. 


Fig. 2. Six frames illustrating weaknesses and strategems in 
programming. 
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Programmers lecture in frames when they are not sure 
what they want to teach, when they have not clearly 
identified the specific objectives of the frame. Analyzing 
this frame in terms of cue, context, and response, we 
see that it is quite possible that a student could answer 
the frame solely on the basis of the last sentence. The 
other material in the frame, representing facts which 
the programmer might want the student to know, neither 
helps produce the response called for nor does it provide 
the kind of context in which we want the response of 
temperature to necessarily occur. In programming, we 
not only provide facts to the learner, but we help him to 
learn. If the student doesn’t understand, it is our fault, 
not his. This is the eighth rule of programming: don’t 
lecture, teach. 

Frame B illustrates another very common error made 
by beginning programmers. Here the programmer had 
the beginnings of a quite satisfactory frame, but after 
he had written out a statement, he then selected a rather 
trivial word to be filled in by the student. The frame 
could be improved tremendously had he left the word 
“liquid” where the blank is now and removed the word 
“expand.” It is the concept of expansion, after all, which 
the programmer was apparently attempting to teach. 
Advanced programmers sometimes make ‘almost an op- 
posite error. They decide that because the word or re- 
sponse itself seems trivial, it is not important enough 
to be left blank. 

Frame C illustrates a frame that calls for such a 
response. Here, the word “not” is important. The fact 
that bismuth and antimony do not contract is the point 
we are trying to get across. The ninth rule of program- 
ming: evoke a relevant response. 

Frame D illustrates another one of the rules. It illus- 
trates how we can frequently depend on rather subtle cues 
to insure that the appropriate response will occur. It was 
not necessary, in this instance, to tell the student pre- 
viously that most substances contract wheit they solidify. 
Instead, we have let the student make his own gen- 
eralization and discover his own principle. The tenth 
tule: don’t provide more cues than necessary. 

Now look at the next frame (Frame E) in Fig. 2. 
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This frame, while not especially well written, does get 
its point across but it also illustrates another important 
error. The programmer may have assumed that his 
students would recognize the relationship between 100 
degrees, boiling water, and the centigrade scale of tem- 
perature. I suspect that if we tried this frame on a 
large number of students, we would find that several of 
them would make the error of inserting the word Fahren- 
heit (if they could spell it) into the blank. This is our 
eleventh rule: don’t assume too much knowledge. 

In the last frame in Fig. 2 (Frame F), we can see that 
the author failed to recognize that we learn from respond- 
ing and not from reading. It is very unlikely that the 
response Celsius would be evoked here in the average 
student unless he had previously learned the astronomer’s 
name. Recalling the preceding frame, we can see one 
possible source for the programmer’s error. He attempted 
to present too many facts within a single frame. The 
twelfth rule is: don’t present two new facts in one frame. 

These twelve rules sum up much of what we know 
about the art of programming at this time. As noted 
earlier, they tend to be incomplete, and we have not yet 
achieved our objective of being able to provide firm di- 
rections to beginning programmers. The lack of definitive 
tules, however, does encourage many programmers to 
experiment and thereby produce new rules and new 
concepts of programming. Perhaps one of the greatest 
errors that a programmer can make is attempting to copy 
the kinds of programs he has seen before. His objectives 
will frequently be too minimal, his program uninteresting, 
and he will fail to produce the kind of learning that he 
feels should occur in the classroom. There is no reason, 
for instance, why the blank in a program shouldn’t 
represent a sentence or two. As another example, there 
is no reason why the programmer can’? employ pictures 
—or for that matter, even have the students draw their 
own. We have used both of these kinds of responses in 
our physics program. 

The last two frames I would like to illustrate both 
tepresent instances where the student is required, in 
terms of the way the frame is structured, to create a 
response which he may never have seen before. 
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The first frame in Fig. 3 (Frame G) is an example of 
this type. Here the word “also” cues the response. The 
student may never have seen the word “Kelvin” before, 
but yet he is able to respond that Kelvin is a scale of 
temperature. This frame also illustrates the use of the 
instructional approach which I have been describing. In 
programming, the primary objective of each frame should 
be the active participation, of the student in terms of 
carefully guided or controlled practice. An auto-instruc- 
tional program should not attempt to explain a prin- 
ciple to a student. Instead it should carefully lead him, 
one step at a time, toward the desired level of proficiency. 
Here, in this frame, we have done more than could be 
accomplished by presenting the fact that “Kelvin is a 
scale of temperature” to a student; we have forced him 
to actually practice the correct response, in the proper 
context, with a minimum of cues. 


Centigrade and Fahrenheit are both scales of tem- 


Fane) g perature. Kelvin is also a* 1... .eceeeeeseceeees 


Knowing that most metals expand when heated, 
and remembering that the period of a pendulum 

Frame H depends on the length of the rod, we would expect 
that a pendulum clock Would ....esesceceseeee . 
time on a cold day. 


* More than one word belongs in this space. 


Fig. 3. Two frames that lead to discovery by the student. 


The next frame (Frame H) illustrates further the 
non-rote quality of learning possible to achieve with an 
auto-instructional program. In a frame of this sort, 
which ordinarily» would have been preceded by steps 
which taught the two facts appearing in the first part of 
the frame, the student is encouraged to discover some 
new fact about the world around him. In nêither this nor 
the preceding frame do we depend on explanations. 
Instead, we have tried, subtly, to guide the student and 


help him experience the satisfaction of being right, all 
on his own. 
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One of the greatest contributions which programming 
is likely to make to education is the clear realization and 
understanding of the learning and instructional processes. 
I suspect that concepts such as the careful control of 
responding to insure its initial accuracy may improve 
classroom teaching whether or not auto-instructional 
programs have been devised. 

What I am saying is that the means for placing 
materials on paper is not as important as our under- 
standing of the instructional process. To illustrate, not 
too long ago I read somewhere of a technique used at a 
school where every effort was being made to teach the 
children cooperation as opposed to competition. While 
this only represents one small part of the training that 
would likely be necessary, I feel it was an ingenious 
solution to provide the children in the pre-school play- 
ground with an assortment of toys of the kind that 
tequired the cooperative activity of two or more children. 
That is, teeter-toters instead of swings, and wagons so 
large that the strength of two children was required to 
move one. This example of programmed learning of a 
social skill or habit illustrates the point that while it 
may not always be feasible, practical, or economical to 
provide instruction independent of a human teacher, 
there seems no reason to believe that if a skill, fact, or 
concept can be taught, it would not be possible to achieve 
it using auto-instructional methods. 
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EDITOR’S NOTE 


Predictably, the first studies of pro- 
grammed instruction tended to be evalu- 
ative; i.e., the purpose of the studies was to 
compare programmed materials and con- 
ventional instruction on their efficiency as 
teaching devices. Quite predictably, for the 
majority of such studies, “no significant dif- 
ferences” was the conclusion. Once that it 
had been determined that programmed ma- 
terials could teach, attention was directed 
toward research which would isolate some 
of the parameters of learning with material 
of this type. Only a relatively small amount 
of data is available at the time of writing, 
but it is sufficient to point the directions 
which research will take in the immediate 
future. Some of the areas for which more 
information is needed are: 


What are the (1) characteristics of the 
“ideal” frame or step? This can be subdivided 
into (a) What are the characteristics of the 
“ideal” stimulus elements? (b) What are the 
characteristics of the “ideal” response element? 
(c) What type of confirmation technique is 
best? and (d) What type of activity will 
mediate best between the stimulus and re- 
sponse elements? 

What type of programming technique zesults 
in the greatest gain, with time to learn held 
constant? 

How we'l is knowledge acquired from pro- 
grammed materials retained? 

Is it necessary to develop programs which 
take individual differences in ability and mo- 
tivation into consideration? 
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Part Ill 


What type and quality of reinforcements 
should be used with programmed instruction? 

What are the best criterion measures for 
evaluating learning from programmed mate- 
rials? 

What characteristics must a program have 
to insure satisfactory transfer of training? 

What are the criteria needed to evaluate the 
quality of a program? 

What learning models, if any, provide a 
satisfactory frame of reference for further de- 
velopments in programmed instruction? 
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Development and Appraisal of Devices 
Providing Immediate Automatic Scoring 
of Objective Tests and Concomitant 
Self-Instruction* 


S. L. Pressey 
Ohio State University 


A. “SELF-SCORING” TESTS: THEIR UNIQUE 
CONVENIENCE, AND USEFULNESS IN INSTRUCTION 


The grading of conventional essay-type examinations 
has long been recognized as burdensome and slow. By 
contrast, the scoring of objective tests has seemed easy 
and prompt. But actually this latter task is by no means 
negligible, or as immediate as might be desired. Objective 
tests are most commonly multiple choice or true-false. 
The person taking the test checks the answers he con- 
siders right, on the test blank or on a separate answer 
sheet. The materials are then collected, and at some later 
time the errors are checked and counted. If test materials 
or simple testing devices could be developed such that, 


*The work described in this paper Was under subvention 
from the Special Devices Center of the Office of Naval Re- 
search, for development of devices for the facilitation of 
learning, worlting in cooperation with the Ohio State Univer- 
sity Research Foundation, under the direction of the writer. 
This article is reprinted from The Journal of Psychology, 1950, 
29, 417-447, and is used here with the permission of the 
author and the editor. 
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as a student answered each question, that answer was 
immediately and automatically scored and recorded as 
right or wrong, then clearly much trouble would be saved. 
Moreover, results would then be available as soon as the 
test was finished. If a future emergency should necessi- 
tate the rapid training of large numbers of men, such 
convenience and immediate availability of scores could 
appreciably facilitate the training. 

Additional advantages of self-scoring devices are more 
important, however. A test in French or mathematics or 
navigation is of little value if it does not in some way 
further the student’s learning of that subject. If he is 
weak on certain points, the test should locate them and 
aid in the remedying of these weaknesses. And this 
should be done promptly; an instructor who never an- 
swers a student’s question until 48 hours after it is asked 
would be considered exasperatingly inefficient. The usual 
testing methods are grossly at fault in all these respects. 
Usually, the student does not find out how he did on a 
test until a day or more after it was taken. Even then, he 
may be given only his total score or grade, without any 
indication as to questions missed. If the scored test 
blanks are returned to him, he may be little or no better 
off. Even if his errors have been marked on the test, he is 
still left in ignorance as to which other answer on a mul- 
tiple choice question is tight. If separate answer slips are 
used and these returned checked but without the test 
blank, he knows for instance that he missed Questions 12 
and 19 but does not know what Questions 12 and 19 
were.1 In contrast, self-scoring devices inform the student 
immediately, when he indicates his answer to an objec- 
tive question, whether that answer is correct; if it is 
wrong, he is at once automatically guided to the discovery 
of the correct answer, This quick incisiveness should sub- 
stantially facilitate instruction. In an emergency, requit- 
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ing the rapid training of large numbers of people, such 
integration of testing and teaching into one episode 
should be especially helpful. 

Devices or special materials which at once inform a 
student about the correctness of his answer to a question, 
and then lead him to the right answer clearly do more 
than test him; they also teach him. And they do this in 
ways so in accord with basic laboratory findings and 
theory regarding the nature of the learning process that 
they constitute, to an exceptional degree, an example of 
human engincering. Theories of learning recognize the 
paramount and obvious importance of the learner’s 
knowing whether each response he makes is correct or 
not. Research has shown that such knowledge is most 
effective if obtained without delay. There are data to 
indicate that answers found to be wrong, but not cor- 
tected, tend to be repeated. Other things being equal, 
the response which has been made most often, and most 
recently, is most likely to be made again. The devices 
suggested above provide that the learner always is in- 
formed about the correctness of every answer to every 
question on a test. That information is given immedi- 
ately. If a mistake is made, the learner is at once guided 
to the right answer. This is always his last or most recent 
answer before leaving the question; and if the test is 
repeated it is almost certain to be made most often. 
Somewhat as the automatic pilot on a plane adjusts the 
flight more quickly and completely than a human pilot 
could do, so such devices spot each learner’s weaknesses 
and assure their correction more adequately and immedi- 
ately than could any human instructor. 

As an instructional device, the self-scoring test has cer- 
tain further advantages. An objective test can cover many 
more questions in a given time than a written examina- 
tion or quiz. The self-instructional objective test can 
similarly cover more ground than would be possible other- 
wise. In a given time, it can in effect give individualized 
instructions to every student in a class (searching out the 
misconceptions of each student and then correcting 
them) over more questions than a tutor could cover with 
one student. A well made objective test has a more care- 
fully considered and exactly phrased right answer than 
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an instructor is likely to give in class discussion; and the 
wrong answers are carefully chosen as the most common 
misconceptions or mistakes. The self-instructiénal pro- 
cedure systematically presents these for consideration and 
then clearly checks them off as wrong. In short, these 
devices should make it possible to cover more ground in 
a given time than a teacher could and do it more ade- 
quately. 

The purpose of the present paper is, briefly to (a) de- 
scribe means by which tests may be made self-scoring and 
self-instructional, (b) investigate the way in which such 
devices work, and (c) determine their value in educa- 
tional or training programs. 


B. MATERIALS AND METHODS 


1l. The “Self-Scoring” Device 

The total project has involved development or im- 
provement of several different devices.2 However, the 
data of the present paper were all obtained with the 
“punchboard.” This is a very simple unit the size of a 
three by five card, about one-half inch thick. A center of 
quarter-inch ply-board has riveted, on either side, two 
thin sheets of pressboard. Between the outside sheet and 
the second sheet is sufficient space for a sli 
be inserted; between the second sheet of the pressboard 
and the ply-board center is a temovable key sheet of 
Pressboard. On each face of the punchboard are two 
columns each of four rows of one-eighth inch holes, the 
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rows of four holes being numbered from one to 30. How- 
ever, the key sheet has holes only for the right answer on 
the particular test to be used. 

Test questions are usually presented on a mimeo- 
graphed sheet, but may be given orally, put on the black- 
board, projected on a screen, Or otherwise presented as 
convenient.-The testee takes a test by simply punching 
with a pencil point through the paper slip in that hole 
which corresponds to what he thinks is the right answer. 
For instance, if for Question 1 of the test he thinks the 
second answer is right, he will punch his pencil through 
the paper appearing in the second hole of the first row. 
If he is right, his pencil goes through the paper and down 
into the hole in the key sheet. But if he 1s wrong, the 
pencil barely breaks the paper and then comes up against 
the key sheet. He thus knows that he is wrong and tries 
another hole in Row 1, thus proceeding until the pencil 
goes deep. 

After he has gone througi: a test, he can glance back 
and see on what questions he made a mistake (where he 
punched more than one hole or has a small hole). This 
can be done either before or after the sheet is withdrawn 
from the punchboard; and either before or after, it is 
easy to count up the number of small holes to obtain the 
error score. A space at the top of the front plate of the 
punchboard gives room on the paper for the testee to 
write his name and other information as well as the error 
score. è 

The device is very simple to use. There is no possibility 
of changing an answer (a hole cannot be unpunched). 
The punched papers, once withdrawn, can be filed and 
are a convenient record for item ‘analysis or other study. 
The keys are so planned that each can be turned over or 
turned end for end, and thus give four different patterns. 
Two or three different key cards thus can total enough 
variations in pattern so that learning a key seems very 
unlikely. In setting up a test, the procedure is to prepare 
the pattern of right answers to correspond with one of 
the ready-made keys, which have been carefully planned 
to distribute the right answers‘oyer the four possibilities. 
However, it is relatively easy to punch a new key if one 
is desired. For true-false or other two-choice tests, four 
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columns can be used on one side. The back side of the 
punchboard, which is made exactly like the front side but 
numbered “2,” either permits a second trial on the test 
to find improvement and give more practice, or provides 
a total of 60 multiple-choice items, 


2. The “Self-Instructional” Tests 


The tests used with the punchboard in the experimen- 
tation to be reported almost all consisted of four-choice® 
objective questions, of which samples will be given 
shortly. Tests which consisted of a series of statements 
to be appraised as true (by punching the first hole) or 
false (by punching the second) were compact and rela- 
tively easy to make, but seemed somewhat less effective 
for instruction. Tests of this same type but with a third 
category of “uncertain” (shown by punching the third 
hole in a row) were tried but were hard to make, since 


f re occasional long ones. In- 
structional tests were found most effective if followed by 


not have four wrong answers y 


is needed, or which elucidates the qu 
alternative answer should conf 
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discussion the same hour. Even with so short a task as 
a 30-item true-false test, several runs through it with the 
punchboard were often found to be needed, if a perfect 
score was to be reached. All these and other considera- 
tions put together led to the conclusion that a self- 
instructional test of only abort 30 or 40 questions was 
better than any longer unit. Moreover, such short tests 
had obvious practical advantages. With a little careful 
planning, a test of 30 four-choice questions could be put 
on one side of a single long mimeograph sheet, or 30 
true-false statements on one side of a short sheet, and 
almost any test of that many items on two sheets or two 
sides of one. In addition, the punchboard could be kept 
small.- For use on the arm of the usual classroom chair, 
such compact convenience was a distinct asset. 

Initial experimentation was with a great variety of 
subject-matter ranging from nonsense syllables, vocabu- 
laries in a foreign language, and glossaries of naval terms, 
to technical articles and chapters from books used in the 
Naval R.O.T.C. training program at the University, 
and series of tests covering the required reading in two 
courses in psychology. It was finally decided to concen- 
trate on material of three types. Russian vocabulary was 
used as réte matter more interesting and valuable than 
nonsense syllables, equally unfamiliar to almost every- 
one, and indicative of possible values of self-instructional 
tests in learning a foreign vocabulary. Hard English vo- 
cabulary items were used to represent tasks such as the 
learning of technical terms where the material was highly 
meaningful but (in contrast to the subject-matter of a 
course) without any organization or structure. And the 
subject matter of two psychology courses served as an 
example of material which was highly organized, and 
part of a systematic program of instruction. The follow- 
ing excerpts from certain of these tests will illustrate the 
simple directions used and some typical items. The psy- 
chology test excerpt includes two features found dis- 
tinctly usefu? for review study after a test is taken: head- 
ings in the test to give it structure, and page references 
so that if a student misses a question and wishes to look 
it up, he can turn at once to the place in the reading 
where that topic is treated. 
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Russian Vocabulary Study Test 

Directions: In each line find the English word meaning 
the same as the Russian word, and punch your pencil into 
the hole corresponding to that English word. If your pencil 
goes deep, you are right, and should go on to the next line. 
But if your pencil barely breaks the Paper, you are wrong, 
and must try again—until you do find the tight answer. 
” ah ZapisKa: (a) plate; (b) expensive; (c) note; (d) 
ridge. 

27. Potom: (a) depth; (b) then; (c) paper; (d) word. 


English Vocabulary Study Test 


your pencil goes deep, you have the tight meaning. If it does 
not, try the phrase you think next most likely to be right. 
Continue until you find the right answer. Then study that 
word so that you will know its meaning, if you are tested on 
gain. 
1. ĪNTEGUMENT; (a) a completed whole; (b) a covering 


as the human skin; (c) moral dness; i 
ears (c) soundness; (d) a factor, as in 


19. : ; i i ial; 
(a) mae (a) favorable; (b) unique; (c) influential; 


Psychology Study Test Abilities 


Directions: Punch the hole indicati i 
ting what you think the 

Pett answer to each question; if wiong, try =n tig 
aa Ability: 12. A person Superior in art usually has 
$e) noor socal adjustment; (b) a good general ability and 
ra ages Gt special Opportunities and motivation; (c) a 
ee ae innate capacity; (d) lack of ability in other 

Appraisal of Ability: ili 
eat Ton Eagla a $ 0 ihe The general ability of a person 
a (a) personal interview; 
test; (d) group irtelligence test, 77-1. 


For some of these te 
using diffe 
im material was sought to 


tests, true-false-uncertain jud ent test, 
and an application test Consisting of a Pak of para- 
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graphs each presenting a practical situation or episode 
followed by questions as to ways of best dealing with 
such a situation. There were also various check tests to 
determine the effects of the practice tests, which will be 
described later as used. 


3. Groups Experimented With 


At the Ohio State University the first and second 
courses in elementary general psychology and the first 
course in educational psychology are taught in sections 
of about 35 students, mostly freshmen and sophomores. 
In a given quarter there may be a total of as many as 50 
sections. Most of these are taught by part-time instruc- 
tors in their third or fourth year of work for the doc- 
torate. Each course is under the supervision of a senior 
staff member; there is a regular calendar, certain objective 
examinations are taken by all sections, and there are fre- 
quent staff meetings. These numerous sections, thus co- 
ordinated and systematically appraised, offered unusual 
opportunity for the extensive group experimentation de- 
sirable in this project. Further, it was possible in the 
course in educational psychology to make sweeping 
changes in methods and hours of meeting when needed 
for experimental purposes. Most of the data are from 
various of these sections. However, students in certain 
other courses, from a how-to-study laboratory for fresh- 
men on probation to certain graduate seminars, took part 
in the total investigation at one time or another. In short, 
for the three quarters of the regular school year, the total 
number of cases, sections, and facilities for this investiga- 
tion seemed exceptional. 


C. RESULTS 


In attempting to appraise the value pf devices involv- 
ing immediate automatic scoring and self-instruction the 
natural first step was to determine whether, in fact, tak- 
ing a test by such means would at once bring about 
learning. A natural next step was to investigate whether 
such devices were of service in regular instructional or 
training programs. A desirable third step was to see 
whether self-scoring tests might be exceptionally useful 
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with special groups or instructional plans. These issues 
wilt be taken up in order.* 


1. Immediate Outcomes of Use of the Punchboard 


As mentioned earlier, Russian vocabulary, hard English 
vocabulary, and subject matter in Psychology (as illustra- 
tive of respectively, rôte matter, meaningful but un- 
organized material, and material organized and meaning- 
ful) were chosen to show how the punchboard worked 
with matter of various types. Certain immediate out- 
comes of such use are now to be considered. 

a. Effect of the punchboard on range of test scores. 
Ordinarily a person taking a multiple choice test checks, 
for each question, the answer he thinks right; and his 
total score is the number of questions answered correctly, 
or answered incorrectly. In interesting contrast, a self- 
scoring procedure such as the punchboard can measure 
degrees of ignorance for each question. For example, on 
a four-choice question a student could make no errors, 
or one, two, or three. On 30 questions, the total possible 
errors could range from zero to 90 as compared with 
zero to 30 for the same test taken in the usual fashion. A 
procedure which increases the range of scores three times 
should discriminate more widely between those students 
who had the subject-matter well learned and those who 
still were grossly ignorant or had many misconceptions 
to correct. 

A very simple experiment suggested that this was so. 
One college class was given a hard English vocabulary 
test in the usual way; each student checked the answer 
to each question which he thought right, and the score 
was the number of questions wrong. Lowest number of 
errors was 18 and highest 28, or a range of 10. Then 
another similar class was given the same test using the 
punchboard. Lowest number of errors was 19 and highest 
53, or a range of 34! It would surely seem that a range 
of 34 would discriminate good from poor students more 
agreeably than a range of 10, or less than a third as 
much. Similarly the range on a 30-question test in psy- 

“Acknowledgments for work on various phases of the total 


study should be made to Dr. Viola Cassidy, Dr. Leslie Briggs, 
Mr. Daryl Severin, and other ascent | eee rie 
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chology was 8 for the usual method of testing, but 33 
using the punchboard. A test of Russian vocabulary, 
given immediately after five minutes study of the 30- 
word Russian-English vocabulary list, showed less differ- 
ence in range (20 as compared with 26) because the 
preceding study had made the test too easy. But the 
difference is nevertheless there. 

b. Effect of use of the punchboard on number of 
errors when a test is immediately repeated. If students 
take a test in the usual way and then at once take it 
again, they would be expected to make about as many 
errors the second time as the first. Table 1 shows that 
they do so. In fact, sometimes they do a little worse the 
second time, as a result perhaps of boredom and care- 
lessness. The first two columns show that when a class 
took a 30-word Russian vocabulary test in the usual way 
(the students simply checking what they thought the 
right answer) and then were immediately given the same 
test again in the same way, the median number of errors 
increased from 7 to 8. But a class which took this same 
test twice using the punchboard, showed a drop from 12 
to 8 errors. With meaningful material the learning with 
the punchboard was even greater: from 43 to 7 errors on 
the English vocabulary test and 18 to 2 on the psy- 
chology quiz (Table 1). 

It seems clear that the students do show some immedi- 
ate learning as a result of using the punchboard. Appar- 
ently they gain more with meaningful material, and least 
on matter of a rôte type such as the Russian. The great 
gain in the English vocabulary might be explained as 
due to the great initial difficulty of these terms plus their 
basic familiarity, making them easily learned, once the 
punchboard gave reminders as to their meaning.® 


€ For this same reason, initial error scores on the Russian test 
in Tables 1, 2, and 3 following are probably lower than they 
should be for comparison with the results in hard English 
vocabulary and Psychology. But important conclusions appear 
not thereby affected. Moet 

° Detailed analysis of the above and other similar results 
indicated that the punchboard brought improvement in two 
ways. When a right answer was at once hit upon, the first time 
through a test, the punchboard confirmed this choice as right. 
But this effect was not very important. It brought gains over 
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TABLE 1 


Learning from taking a test with the punchboard, 
as shown by decrease in errors on second testing. 


Subject matter of 


Resta? Russian English Psychology 
Method of giving As Punch-| As Punch-| As Punch- 
tests: Test board | Test board | Test board 
Median? number of 
errors 
On first test 12 22 42 11 18 


7 
On second test 8 


Decrease in errors -1 


© The median is the point below and above which there are equal 
numbers of cases. Classes averaged 31 students in size. 


c. Effects of the punchboard as evidenced by a test 
different in type from the practice tests. The above re- 
sults were obtained with immediate repetition of the 
same test. Conceivably, learning brought about by the 
punchboard might be highly specific—perhaps a mere 
memorizing of the test material, or the key. If this were 
true, a second test different from the practice material 
would show little gain. The following simple experiment 
offers evidence that the learning was not so limited, that 
there was instead a real gain in knowledge of the matter 
presented. 

The multiple-choice Russian test used previously was 
first given a class, using the punchboard. Then a very 
different test, a recall test, was given. On this recall test, 
the Russian words were listed in a different order from 
that in which they appeared on the practice test, and a 
space was provided after each Russian word for the stu- 
dent to write its English meaning. This same recall test 
pares < 


what occurred simply with repetition of the test as a test, of 
only about 5 per cent. The big effect of the punchboard was 
in correction of errors. The punchboard brought immediate 


right answer on second trial for ab ‘tems 
wrong on the first testing, about 65 për cent of item 


 ——— 
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was given also to another similar class, after the practice 
test had been taken simply as a test. However, the order 
of items was different. And the “direction” of the think- 
ing was reversed. In the check test, a definition was pre- 
sented and the student indicated which of four difficult 
English words fitted the definition. In contrast, the prac- 
tice test presented the word and the student chose the 
correct one of four definitions. The Psychology check test 
differed from the practice test in an even more elaborate 
fashion. Ten questions on one general topic did not ap- 
pear in any form on the practice sheet; 10 were para- 
phrased from questions on the practice test; the remain- 
ing 10 were repeated from the practice series but ap- 
peared in a different sequence on the test, intermixed 
with the paraphrased and new questions, and with the 
order of the wrong answers and position of the right 
answers changed. This scheme was used extensively later 
in this study. Its purpose was to obtain evidence regard- 
ing the spread of punchboard learning as affected by the 
degree of similarity, in phrasing and idea, between 
the practice and the check tests. Table 2 summarizes the 
results of these check tests, given immediately after the 
corresponding practice test had been taken with the 
punchboard. 

Effects of the punchboard are here not so great or so 
clearcut as in Table 1; a repetition of exactly the same 
test would inevitably show more gain than check tests 
which vary from the instructional test. However, on the 
English and psychology the differences are substantial 
and (as indicated by the note to Table 2) distinct’ 
Analysis of the four-point gain in psychology showed 
three on the “re-arranged repeat” items and one point 
on the paraphrased—an analysis here of little reliability 
but later to be further investigated. 

d. Effects of several consecutive repetitions of a test 
using the punchboard. If once through a practice test 
with the punchboard does bring some learning, several 
times through. should bring more. However, if students 
go through the same self-instructional test several times 
in succession, it is conceivable that they might tend to 
learn the key or pattern of right answers rather than the 
information conveyed in the test. Or at least, they might 
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TABLE 2 


Learning from taking a test with the punchboard, as shown 
by number of errors on a second test different in form. 


Subject gatiero Russian English Psychology 


Method of giving As Punch-| As Punch-| As Punch- 
first test: Test board | Test board | Test board 


Median number of 
errors on second 
test 19 17 9° 2 10° 6 


* Only 5 per cent of the class taking the English practice test 
simply as a test, and 12 per cent of those so taking the psychology 
practice test, did as well as or better than the median of the punch- 
board group. Table 3 following shows further reduction of Russian 
recall test errors to 10 after 5 punchboard testings. This is addi- 
tional evidence that the practice tests were causing a learning 
which was not confined to mere rote learning of the practice test 
series. Again, these classes averaged 30 students in size. 


learn that information so much in one particular order 
and context that use in other setting would be handi- 
capped. To check on these points, one class was given the 
same Russian vocabulary test practice test four times 
using the punchboard, and then a recall check test, all 
in one class hour. A similar class was then given a series 
of practice tests covering the same words but with the 
order of items and the key or pattern of right answers 
changed each time; again the punchboards were used but 
changed each time to fit the key for that particular test. 
And again at the end of the series the recall and test was 
given. Similar series were run with same and changed 
tests in English vocabulary and psychology except that 
the length of the hour permitted only three times 
through the practice series, and left no time for an end 
test in psychology. Table 3 shows the median number 
$ pos on each practice test and in the Russian check 
est. 

On the Russian, the changing practice tests seem actu- 
ally to handicap the learning, in that the median number 
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of errors actually goes up on the second trial, and there- 
after continues higher than on the tests in which the 


TABLE 3 


Decrease in number of errors with repeated taking of a practice test 
using the punchboard, and effects of changing the order of questions 
and the pattern of right answers upon such decrease.* 


Russian Vocabulary English Psychology 
Practice Practice Practice 
Test Check | Test Check Test 
1 2345 Test | 1 2.3 ‘Test | 2 G255 


Median No. 
of érrors 
Same 
test 11 853 10 |42145 6 1741 
Changed 
test |1315996 10 |4178 5 18 3 1 


* Numbers of cases for Russian groups were 49 and 28, for Eng- 
lish 29 and 41, and for psychology 33 and 34. Error scores on end 
tests are mistakes plus omissions. Five minutes study of the Rus- 
sian vocabulary list preceded the practice tests. 


order was unchanged. However, the check test shows no 
difference, at the end of the practice series. On the Eng- 
lish vocabulary and psychology, results for same and 
changed practice material are practically the same. 

A natural interpretation of the above results is that 
with such réte learning as of the Russian, there tends to 
be some dependence on the arrangement of items. How- 
ever, in proportion to the meaningfulness and integration 
of the material, change or sameness of key makes no 
difference; attention is on the subject-matter, and that is 
what is learned. Apparently with none of these subject- 
matters does repetition of the practice test with the same 
key handicap learning. It is even conceivable that the 
same test and key in successive reviews might help. For 
example, in reviewing a book it is an aid that one can 
each time find a topic on the same page with same con- 
text, and upsetting if a review is of a different edition 
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where all these details are different. The analogy is of 
course only very rough, but it does seem suggestive of 
value in review of the same test. Some value would seem 
especially likely if the test questions were in a logical 
order and perhaps grouped under headings, as suggested 
a few pages back. Anyhow, these findings are indeed 
pleasant, since they support the more convenient repeti- 
tion of the same test for practice. They also thus warrant 
use of a specialized device, for facilitating repeated runs 
through a practice test, to be mentioned later.” 

In short, the punchboard does indeed appear to further 
learning. It is probably more effective with meaningful 
than rote matter. Repeated use seems to carry learning 
further; and some repetition of the same test does not 
seem to cause mere learning of a particular test key or 
other unfortunate effect. However, the results so far have 
been with brief and relatively simple tests and checks on 
outcomes, all within one class hour. The practical ques- 
tion now is whether these procedures would be effective 
yn more substantial learning tasks, and over a period 
of time. 


2: The Punchboard, in a College Course, as 
a Testing Aid Not Integrated with Teaching 


The most simple but obviously inadequate way to use 
a device like the punchboard is as a convenience in test- 
ing but without any attempt to tie its instructional po- 
tentialities in with teaching methods. Such a situation 
might occur if its use were required of teachers without 
interesting them in it or indicating ways of using it to 
aid instruction. Two trials of what might be called per- 
functory use were made, 
o of punchboard tests for review. In the large 

-quarter course in general psychology, eight cha 

ters of the textbook wate coveted. Teren hE fit d 
second midterm examinations. To help students in re- 


* Readers familiar with the method i 
k s of Thorndike and others 
in me nay of the law of effect will recognize shat repetition 
a a tes ane the punchboard involves the “retained situation,” 
i problem being “retained” before the learner until the 
Tig tanyer is found. It would seem that such devices as the 
pane oard could be very useful in such research. Results of 
such theoretical investigation will be reported in a later paper. 
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viewing for the second examination, two 60-item review 
tests were constructed. The first consisted of 20 multiple- 
choice questions on the first two chapters, 20 on the 
second two, and 20 more which were the first 20 repeated 
but in a different order. This review test was given to 
four sections totaling 114 students during the class hour 
preceding the midterm examination. A second review test 
similarly covering the second four chapters was given to 
six other sections, totaling 182 students. On the day fol- 
lowing, all sections were given an objective midterm 
examination, having 21 questions repeated or para- 
phrased from the first review test, 22 similarly from the 
second practice test, and 24 questions which were on the 
same eight chapters but unlike questions on either review 
test. 


TABLE 4 


Percentages correct, of mid-term questions covered in a review practice 
test using the punchboard, and of other questions not thus reviewed. 


Group A | Group B 


Questions not in practice tests 16% 75% 
Questions in practice test given Group A 85 69 
Questions in_practice test given Group B 70 79 


Table 4 shows that on these new questions the two 
groups did practically the same, thus indicating that they 
were essentially equal in ability and accomplishment 
aside from any help the practice tests might have given. 
On the midterm questions which were repeated or para- 
phrased from the first practice test the group which had 
taken it did distinctly better; similarly, the second group 
did better on questions from the review test it took. The 
differences are not great, but are significant. Table 5 
indicates that use of the punchboard reduced errors most 
on the repeated questions. But some gain appeared also 
on those which had been rephrased from the practice test 


to the midterm. ie: à 
In the above experiment, no significant differences 
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appeared between questions which appeared once and 
twice on the practice test. Initial difficulty was found 
very important, however. Obviously, the punchboard 
could bring no more than 5 per cent improvement on a 
question so easy that 95 per cent of the students at once 
answered it correctly on the practice test. On most of the 
difficult items, the device caused large gains. However, 


TABLE 5 


Per cents of students getting questions right on midterm which they 
(a) had taken on a practice test using the punchboard, and 
(b) had not taken on a practice test. 


Not Taken | Gain From 
Taken in With Punch- 
Practice | Punch- board 


Test board Test 


Questions paraphrased from 


practice test to midterm 69 72 3 
Questions repeated from prac- 
tice test to midterm 69 87 18 


The table shows that 72 per cent of the students got questions 
right on the midterm which they had had in different wording on 
a practice test using the punchboard. However, only 69 per cent 
of students got these questions right if they had not taken them 
in the practice test. The punchboard practice test thus brought 
a gain of 3 per cent; in repeated questions the gain was 18 per cent. 


on a few paraphrased questions, the Practice test actually 
caused loss, presumably because of tricky rephrasing. 
b. Value of frequent brief practice tests using the 


punchboard. In the next quarter, trial was made of the 


possible value of frequent brief testing using the punch- 
board, in the second month of the course in general psy- 
chology, in which eight chapters of the test were covered. 
Four 20-question practice tests were made up, each sam- 
pling points covered in two chapters. On the day when 
discussion of one of these two chapter units was to begin, 
five sections totaling 149 students took the practice test 
on that unit using the punchboard. Afterward, any ques- 
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tions about the test were answered; but since the instruc- 
tors regarded the testing as an experiment in which they 
had no hand, they did not encourage discussion nor use 
the test in their plan of teaching. Five other sections, 
totaling 152 students, on the same days took these same 
practice tests in the usual fashion of test-taking; they 
simply marked on the test sheet what they thought to be 
the right answers. The papers were then collected, graded 
after class with all errors marked, and returned the next 
day. At this time the instructor went over the questions, 
indicating the right answers, and giving opportunity for 
discussion. This procedure might be considered to lean 
over backwards in the effort to assure full demonstration 
of the values of frequent test without the punchboard. 
A total of more class time was given to the tests than to 
the punchboard procedure. The tests were seen twice, on 
two different days. Probably in most classes in which fre- 
quent brief tests are given, the scored tests are returned 
less promptly, and without such review of questions and 
right answers. A third group of five sections covered the 
same ground, but without taking the practice tests in any 
way. Most of the instructors had two sections; and as far 
as possible it was arranged that each had one of one type 
and one of another. 

Evidence as to the initial status of the three sets each 
of five sections before the experiment began was yielded 
by the first midterm examination, since all 15 sections 
used the same instructional procedure during this first 
month of the course. The first row of figures in Table 6, 
giving the average per cent of right answers made by each 
group on this first examination, shows the “no practice 
test” group initially best and the punchboard group 
poorest. On the second midterm the two groups which 
have taken the practice tests are slightly above the first 
group. Analysis shows the punchboard group best on the 
25 questions repeated and 15 paraphrased from the prac- 
tice tests, but poorest on new questions—as might be 
anticipated from their initial bottom position (Table 6). 

Occasional brief tests not closely tied in with instruc- 
tional methods thus brought only slight gains. Tests 
given in the ordinary way, scored after class, then re- 


turned and discussed the next day, were about as effective 
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TABLE 6 


Percentages of questions answered correctly by three groups of students 
(a) on the first midterm, before the experiment began, (b) on the 
second midterm, and (c) on each type of question in the second 
midterm. 


Groups of Students 


No Tests 


Practice | Taken Tarea 
Tests as Punch- 
Taken Test board 
First midterm 69 68 66 
Second midterm 73 78 76 
28 questions not in the prac- 
tice tests in any form 73 75 69 
15 paraphrased questions 77 82 83 
25 repeated from practice 
tests 71 81 83 


* Five sections in each group, the first totaling 149 students, 
the second 152, and the third 149. 


as the punchboard. However, it did save the work of 
scoring, also the trouble and time of returning answer 
slips the next day and going over test questions and right 
answers with the class then. But it would seem that tests 
should not be merely occasional episodes incidental to 
the instruction, but rather be closely integrated with the 
instruction method. What might the gains then be? 


3. Use of Practice Tests With the Punchboard 
as a Major Instructional Method, by the Instructor 
in Charge of the Class 


This experimentation was in the course in educational 
psychology, which follows one in general psychology, is 
most commonly taken in the freshman year, js taught in 
sections of about 35 students, and meets five times a 
week for one class hour. For this work, a large number of 
practice tests had been carefully prepared, with special 
emphasis on problems of judgment and application. 
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a. Value of intensive use of punchboard tests as com- 
pared with carefully planned discussion by the same in- 
structors. First, what might be accomplished by using the 
punchboard as a major instructional device, on certain 
units of the course? These units or topics (as, transfer 
of training) were in the course calendar allotted two or 
three class hours. On the first day assigned to such a 
unit, the experimental sections took a 30-item practice 
test using the punchboard, at the beginning of the hour. 
When a student finished the test, he looked up any 
doubtful questions, and discussed these with other stu- 
dents and the instructor. Since it was understood that 
the practice tests would not be used directly as a basis 
for grading, but rather as an aid toward preparation for 
midterms and final examinations, both instructor and stu- 
dent sought help from the practice testing. A student’s 
total classroom work, including his attitude and effective- 
ness in use of the practice tests, did figure in the larger 
appraisal of him by the instructor, however. After issues 
raised by the practice tests were disposed of, further dis- 
cussion took such direction as the situation to date sug- 
gested. This was to fill any. gaps remaining in considera- 
tion of the subject, and to generalize and apply the mate- 
rial as a safeguard against the undue specificity which 
the objective study tests might otherwise foster. Then 
near the end of the second day assigned to a topic; a 
second practice test was given, with a little time follow- 
ing that for discussion. Finally, the day after the topic 
was finished, a 30-item check test was given in which 10 
questions were repeated from one or the other of the 
practice tests, 10 paraphrased, and 10 new. Of the total 
18 topics of the course, six scattered through the quarter 
were handled in this manner. The few minutes for the 
check test thus always came from a unit not given the 
special treatment. ʻo 
The above procedure was used in two experimental 
sections taught by two experienced instructors who each 
had a control section, where as much time was allotted 
to discussion as was allotted to testing in the other sec- 
tion. In their control sections, these instructors were free 
to stress points which had come out in the punchboard 
groups oF apply other helpful orientations obtained from 
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the experimental procedures; except that the punchboards 
and practice tests were not used on the six units. The 
remaining 12 topics of the course were taught by infor- 
mal discussion in all sections. 

Selection of the two experimental and two control 
sections, from the total of all sections, was by chance 
circumstance. Their status at the beginning of the ex- 
periment was determined by an index taking account of 
each student’s standing on the test of general ability 
given at entrance, his academic record to date, his grade 
in the preceding course in general psychology, and his 
score on a pre-test given at the beginning of the educa- 
tional psychology course and anticipating the major 
topics of that course, to measure pre-knowledge. The 
experimental and control groups were found essentially 
equal initially, as well illustrated by the pre-test means 
in Table 8 to be presented shortly. 

The first analysis was of the effects of the experimental 
procedures on the check tests given the day after each 
unit. Table 7 shows the percentages of questions correct 
in each group for all six check tests combined, or a total 
of 60 questions in the check tests which had not ap- 
peared in any form in the practice tests, 60 paraphrased 
for the practice tests, and 60 repeated from them. Analy- 
sis of variance indicated that in total the differences even 
on the “new items” between experimental and control 
gtoups could be considered significant. It appears that 
when such a diagnostic test-teach device is used as an 
integral part of instruction, results are substantial, and 
they seem to bring better understanding of the topic as 
a whole, so that even “new’ questions in that topic are 


It might be considered that the check tests were given 
too soon after a topic had been completed, and were too 
closely tied in withthe practice tests. Table 8 shows the 
Tesults of tests given later, having no such tie-in, and 
providing much broader coverage. In total, these covered 
all 18 topics in the course, were made up independently 
of the practice tests, and included an extensive case study 
test, unlike all the other material. Pre-test scores illustrate 
findings with a measure of initial status, Means on the 
pre-test are practically the same for contro] and punch- 


a 
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TABLE 7 
Percentages of questions correct on end tests, in two classes using 
punchboard tests as a regular part of instructional procedure, and in 
two control groups taught by the same instructors which took no 


practice tests.2 
No Punch- 
Practice P board 
Test oan Gain 
60 Questions not in practice 
tests 54 64 10 
60 Questions paraphrased from 
practice tests 58 75 17 
60 Questions repeated from 
practice tests 60 83 23 


* The two sections using no practice tests had a total of 57 stu- 
dents and the punchboard sections 62. 


board groups, but on the midterms plus final the punch- 
board group is clearly superior, the difference surpassing 
the 1 per cent level of confidence.8 

As in the appraisal of the review test in general psy- 
chology reported earlier, the design of the above experi- 
ment combined values of the punchboard and the prac- 
tice tests. The combination was here also justified on the 
ground that the device so facilitated the frequent testing 
that values of the tests might be here included as part of 
the total value of the punchboard. However, evidence 
was again desired which would differentiate the effective- 
ness of the practice tests with and without the punch- 


® As already mentioned, two instructors were involved in the 
above experiment, each teaching one control and one experi- 
mental section, so that any differences in their effectiveness 
might affect both groups. Comparisons of the means of the 
groups taught by the first instructor and the second showed no 
significant difference. The inference was that on the whole the 
two instructors did not substantially differ in the groups they 
had to work with, or in their efficiency. Analysis of variance 
also showed no significant difference between the instructors 
in the relations between their control and experimental groups; 
the practice tests with punchboards seem to have been used 
by them with about the same effectiveness. 
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board. The set-up now to be described was planned to 
take account of this issue. } 

b. Value of frequent brief tests with and without 
the punchboard. The plan was simple and straightfor- 
ward. In the spring quarter there were 13 sections of 
educational psychology. Four of these used practice tests 


TABLE 8 


Average scores made by 2 sections who had not taken any practice 

tests and 2 other sections who had taken practice tests on 6 out of 18 

topics, on (a) a pre-test covering essential background and major 

points given at the beginning of the quarter and (b) two objective 

midterms and a final examination given during and at the end of 
the quarter. 


No Punch- 
Practice board | Difference 
Tests Tests 
Mean score on pre-test 45.2 45.6 4 
Mean of midterms plus final 286.7 298.8 12.1 


* There 
in the first group, and 62 in the Punchboard group. Sigma for the 


total midterm plus final score of the first group was 18.9 and of 
the second 16.6. 


punchboard. Two 


5 z n these units, but 
instead of using the Punchboards they marked their 


tests were then collected, and the answer slips graded 
and returned a day or so later, As before, this method 
probably leaned over backward in an effort to give full 
opportunity for advantages of the practice tests without 
the punchboard to be obtained; tather more time was 
given to test-taking plus discussion Of the tests than with 
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punchboard group, and return of the answer slips was a 
second reminder of the test and opportunity for discus- 
sion of them. Seven sections did not use these practice 
tests in any way, the time going instead for informal dis- 
cussion. Initial status was again appraised on the basis of 
an aggregate index taking account of ability as tested at 
entrance to the University, scholarship to date, and grade 
on the preceding course in general psychology. Again the 
three groups were found initially so familiar that no al- 
lowances for initial differences were needed. 

During the first four weeks of the quarter, five units or 
topics of the course were covered. The punchboard 
groups began consideration of two of these with a punch- 
board practice test which was used as a point of depar- 
ture for discussion. They then took a second practice 
test on the second day, in the same fashion as the. punch- 
board groups in the previous project. The test groups 
took the practice tests on these units in the usual paper- 
and-pencil fashion. The remaining sections covered these 
two topics entirely by informal discussion. And this last 
method was used by all 13 sections to cover the remain- 
ing three units. A 75-question multiple-choice midterm 
was then given all sections. Of these questions, 25 were 
tepeated or paraphrased from the practice tests, 10 were 
new questions but on the same two units as dealt with 
in the practice tests, and 40 were questions on the three 
units not having practice tests. In similar fashion during 
the second four weeks of the course four units were con- 
sidered, on two of which two practice tests were given to 
the punchboard and test groups. The second midterm 
was then given. It had 24 items repeated or paraphrased 
from the practice tests, 16 new questions on the same 
units, and the remaining 35 on the two topics which were 
new to all groups. g 

The bars in Fig. 1 show the scores made by the middle 
half of the students in the three groups, on these two 
midterms. The total scores from such varying types of 
questions are obviously the undifferentiated sum of vari- 
ous possible effects of the practice tests. But they seemed 
worth considering as evidence of outcomes in what might 
well become a common situation: practice tests used for 
some but not all the material in a course, and examina- 


136 IMMEDIATE Automatic Scorinc oF TESTS 


tions which appraise students’ work as a whole without 
differentiating portions so aided. The differences between 
the no-practice-test group and the punchboard groups are 
evident and clearly significant. Superiority of the punch- 
board sections over the sections taking the practice test 


FIRST MIDTERM SECOND MIDTERM 
No Pr. As Punch~ No Pr. As Punch- 

s _Test Test board Test Tost board = g 
c 

Ẹ T 5 
€ 6 
60 60 
55 55 
50 50 
45 45 

Maon< 
40 40 


Fig. 1. Scores made of two midterms by three groups of stu- 
dents, those who had taken (a) no practice tests (b) practice 
Hee ey as tests, and (c) practice tests using the punch- 

oard. 

There were 225 students in the seven sections not taking the 
practice tests, 70 in the two sections taking them simply as tests 
but with discussion after, and 132 in the four sections taking the 
practice tests with the punchboard. The bars show scores made 
by the middle half of the students in each group (iiom the 25 
to the 75 percentiles). The practice tests covere: only about 
half of all the topics included in the midterms. 


without the punchboard is evident, especially on the sec- 
ond midterm. As always, individual differences in all 
groups are great. 


Analyses of results are shown, in Table 9, for all the 
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TABLE 9 
Percentages of students who had taken (a) no practice tests, (b) practice 


tests simply as tests, and (c) practice tests with the punchboard? who 
answered correctly various types of midterm questions. 


i Punchboard 

Relation of Midterm to Practice ‘Tests Gains Over 
Practice Tests Not As Punch-| No As 

Taken Test board | Test Test 


First Midterm: Per Cent Correct 

10 Questions not in the prac- 
tice tests in any form, and 
on different topics 63 ól 68 5 7 
10 Questions not in practice, 
tests in any form but on 


same large topics 52 60 62 10 2 
20 Items paraphrased from 
practice tests 66 74 17 11 3 
5 Items repeated from the 
practice tests 77 "90 91 14 1 
Second Midterm: Per Cent Correct 


10 Questions not in practice 

tests in any form and on 

different topics 78 80 83 5 3 
16 Questions not in practice 

tests in any form, but on 


same topics 65 7 78 13 6 
14 Questions paraphrased 

from practice test 68 74 76 8 2 
10 Questions repeated from 

practice tests 73 84 89 16 5 


© There were 225 students in the 7 sections not taking the prac- 
tice tests, 70 in the 2 sections taking them simply as tests but with 
discussion after, and 132 in the 4 sections taking the practice tests 
with the punchboard. 


umn the gain of the punchboard sections over the “test 
control” sections. 

The fourth column shows substantial gains of the 
punchboard group over the group which took no practice 
tests. On repeat and paraphrased items, the gains are 
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greater than those for use of punchboard not integrated 
with instruction presented earlier in Table 6. In addition, 
the punchboard sections show superiority on new ques- 
tions dealing with the same units, and even on questions 
on the units not covered by the practice tests. Apparently 
the discussion and teaching tied in with the tests, and 
aiming at generalization and application, had so oper- 
ated. Moreover, the right-hand or fifth column shows 
some superiority of the punchboard over the test-control 
group; and this extends also, on both midterms, to ques- 
tions on topics not covered by the practice tests. To what 
extent these last differences may be considered significant 
is not clear, but their appearance in all categories surely 
warrants added confidence in them. And again it must be 
remembered that punchboard testing was also more con- 
venient, time-saving, and interesting to the students than 
testing in conventional fashion with discussion the next 
day.® 


° The question might be asked: “Were the punchboard sec- 
tion instructors better”? Their “control” sections the previous 
quarter had been clearly below their own punchboard sections 
and at the average of other instructors; the quarter before that, 
before the experiment began, their sections had been no better 
than the others on midterms and finals. Might superiorities of 
both test groups over the control be due merely to greater habi- 
tuation to tests? Present-day college students are used to tests; in 
the previous course in general psychology they are regularly used. 
Better scores on questions repeated or paraphrased from the 
practice tests would be expected, from students who had taken 
them, but should not be wholly discounted as due to undesirable 
rte recall. After all, it is good instruction to bring up important 
points in class discussion, and sound testing that these points 
should also be covered in any later tests. And the gains on topics 
not covered by the practice tests remained to be explained. 
Apparently the punchboard tests, when integrated with instruc- 
How bronst, more acetate coverage and generalizing of sub- 
ject- han usual instruction— 
ie onventent testing without ec m—more even than the less 

oubtless there should have been a greater , j 
and appraising test material. But the e E EA 
judgment tests, and a unique case study test. Earlier investiga- 
tion under the writer's direction, with a basically similar device 
for self-instructional testing, had used Ree Talss be tice ie ts, 
appraised outcomes with multiple-choice and EE ne 
tions, and obtained similar positive results (Little. 1934). In 
total, t e range in materials and methodology seemed consid- 


S. L. Pressey 139 


c. Student attitudes toward the punchboard. The 
above findings were the results of objective testing. Indi- 
cations of students attitudes toward the punchboard 
were sought. Their liking for it was well evidenced by 
their protest when, after its use part of a quarter, the 
experimental design called for its discontinuance. They 
found the device sufficiently helpful that they came to 
depend on its immediate appraisals, and guidance to right 
answers; thereafter, tests taken in the usual way, and 
leaving them uninformed in these respects, were frus- 
trating. When opportunities were given to take a test 
over again, or to take other punchboard tests on the same 
topic (several were available on each) many students did 
thus extend their self-instruction. Punchboard tests were 
obviously a convenience both to students and instructors, 
for such use. Comments from students were invited. The 
following samples indicate something as to attitudes, and 
methods of using. 


Every time I punch a wrong answer, I read the question 
over again. The punchboard makes me think. 

When I have the right answer, but am not certain, the 
punchboard assures me; then I can settle on that answer and 
remember it, and consider the reasons why it is right. 

Direct questions help me; I don’t retain as much from 
general discussion. 

My errors are pointed out immediately; I’m not left with 
the wrong ideas. 

New ideas and angles are brought out and underscored. 

It’s like a game and a challenge; I’m motivated more. 

_ If you punch a wrong hole, you're the only one who knows 
it; it’s not upsetting like a mistake in class discussion. 

With the series of questions and all the time finding out 
how much I know, I can concentrate much better than in 
ordinary study. 

After I’ve been through a punchboard test, I feel much 
more confident, know where I stand, have been checked up. 

I find it much easier to ask for help, Since the punchboard 
brings issues to a head. 


In short, use of an occasional punchboard test some- 
what perfunctorily as a convenience in testing brought a 
little specific gain in learning. However, use which was 
tied in with instructional method brought decidedly 
more, and spread that gain so that general understand- 
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ings were bettered and study methods made more effec- 
tive. Further, under these circumstances, students liked 
the device and appreciated it as helpful. 


4. Use of Punchboard Practice Tests with Special 
Groups of Superior Students 


It might be assumed that a device facilitating imme- 
diate understanding of test material and self-instruction 
would be most effectively used by the ablest students. 
Since a class, like a fleet, tends to travel at the rate of the 
slower members, any scheme which teleases the abler 
from that restraint should be valuable. Three methods of 
using the punchboard tests with groups of superior stu- 
dents were therefore tried. 

a. The “Accelerate Seminar.” An accelerate seminar 
was tried four different quarters. Selection of students for 
the seminar was based on a brief student data sheet filled 
out the first day of the course in educational psychology. 
This sheet covered age, academic record, veteran status, 
part-time employment, and interest in the seminar as 
briefly described to the group. Those students who were 
interested, whose academic tecords and scores on the 
entrance test of general ability were high, who were not 


evening) instead of the usual five one 
the regular readingş and projects were covered and in ad- 


~~ 


S. L. Pressey 141 


slip and key so that they could do everything themselves. 
They could thus use the tests for self-appraisal and guid- 
ance in small-group discussion, at certain class times and 
also at other hours in the afternoon when the file was 
open to them. Part of the class seminar hour was given 
to special projects or other undertakings and to informal 
discussion at a mature “seminar” level. And since class 
time was saved, each student was supposed to take an 
extra course, work part-time, or make other good use of 
less demand as to class attendance. Median course load 
was 18 hours, and median hours of part-time employ- 
ment 10. 

To determine outcome, students were given essay ex- 
aminations upon the extra reading, were graded by the 
instructor on projects and participation in class discus- 
sion, and took the two objective midterms and final given 
also to the regular sections. The first rows of Table 10 
summarize the results in terms of grades on these objec- 
tive tests made by the regular sections, the accelerate 
seminar students, and cases in the regular sections paired 
with the seminar cases as to age, sex, point-hour-ratio 
through the previous quarter, and grade in the previous 
course in general psychology. With all these categories, 
pairing was a bit rough, but was reasonably satisfactory. 

High (A and B) grades in seminar were over twice as 
numerous proportionally as in the regular sections, and 
failing (E) grades rare. Moreover, the seminar students 
were better than the paired controls, who had over twice 
as much class work. There were more intangible gains; 
many seminar students spoke of increased capacity for 
independent work, and closer relations with other stu- 
dents and the instructor because of the codperative in- 
formality. The instructor found the seminars very stimu- 
lating. It was concluded that two seminars might be 
taught as easily as one regular section, making a net gain 
in number of students handled. Obviously many factors 
were involved. But it seemed clear that superior and well 
motivated students, taught in mature fashion, and with 
such aids to self-appraisal and review as the punchboard 
tests, could save class time and do more than the usual 
total of college work. Moreover, the demands of this 
course did not handicap elsewhere; a follow-up of these 
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TABLE 10 
Grades made by superior students using special instructional methods 
including punchboard tests, as compared with grades of students using 
conventional methods of instruction. 


Percentage of 


4 ber - 
Students Obtaining Ne 
AandB dents 
Grades E. Studep 
— 
27 regular sections 28 11 801 
4 accelerate seminars 65 2 106 
Cases paired with seminar cases 55 5 106 
2 examination-credit groups 96 0 24 
Cases paired with above cases | 83 0 24 
2 self-instructional laboratories 67 0 24 


“A and B grades are high, but Æ is failing. Grades here given 
were based on two objectives midterm tests plus an objective 
examination including a case study test. 


students and their control groups about a year later 
showed the seminar group slightly more superior in aca- 
demic record than before. And as a result of an extra 
course at the time of the seminar plus extra credit ob- 
tained at some other time, a considerable number of 
these students graduated a quarter or more sooner than 
would have been possible otherwise. 

b. The examination credit group. Might superior and 
well-motivated students be put yet more on their own, 
with yet more guidance and help from the self-instruc- 
tional tests? Two “examination credit” groups gave an 
affirmative answer to this question. University rules pro- 
vide that a student may obtain “examination” credit for 
a course without. taking it, if he obtains a grade of B or 
better on an examination covering the course. The ques- 
tion was as to whether superior students could, if given 
a little guidance in independent study plus he help of a 
series of punchboard tests covering the course pass the 
educational psychology without any Tegular class work. 

For this little investigation, records to date of students 
in the pre-requisite general elementary Psychology were 
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Tun over before the middle of the quarter, to find stu- 
dents doing exceptionally well in that course and other- 
Wise superior. The best of those thus discovered were 
interviewed to verify as to their superiority, explain the 
plan, and see if they were interested. Only those strongly 
Interested and clearly superior were finally included. 
Average percentile on the entrance test of general ability 
was 92, average grade to date was high B (a PHR of 3.35) 
and average age 20.5; preference was given to veterans. 
One quarter, nine students were chosen, and another 15. 

As soon as each group was selected, near the middle of 
the quarter, the first of a series of four two-hour meetings 
was held. In these, procedure was further explained, as- 
signments made in large blocks each covering about a 
third of the course, certain laboratory projects briefly 
demonstrated, and two objective midterm tests and a 
final given; there was time for a very little orienting dis- 
cussion. In addition to the four meetings, the total series 
of punchboard tests was made available during about four 
hours each week with at least two hours’ attendance re- 
quired. An additional 20 hours each week was designated 
as office hours when an instructor or assistant was avail- 
able for consultation and punchboard tests were also 
available. Work was thus largely independent study but 
with frequent punchboard test check, and opportunity for 
individual consultation. Often two or more students were 
in for the check tests at the same time, and worked 
together. — 3 

In the row labelled “examination credit group” of the 
above table, results are shown in terms of objective mid- 
terms and finals given regular sections also. The following 
tow gives results for cases paired with those in this group 
as to ability and previous work, in the same fashion as 
for the accelerate seminars. Again results are excellent as 
compared with regular sections, but -not quite as good 
as for the paired cases. Perhaps this indicates the students 
were approaching the limits as to desirable Proportion of 
independer.t work.1° 


1 Neither accelerate seminar nor proficiency group were 
seri TERE Worcester (1945), Munroe (1926), and 
Pa rath d (1935) have reported successful trial of special sec- 
tions meeting less time a week, for superior students, in several 
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c. The self-instructional laboratory. A third venture 
attempted even greater flexibility. At the beginning of 
two different quarters, small groups of very superior and 
mature students were given the opportunity to take the 
course in educational psychology in a “self-instructional 
laboratory.” Each student was given a list of all the work 
in the course, including the usual informal laboratory 
exercises. Again a third more reading was asked of the 
special groups. Five afternoons a week, a special labora- 
tory room was made available to these students, in which 
all the materials and readings for the course and a com- 
plete series of punchboard tests covering the course were 
provided. In charge was an assistant who was an experi- 
enced teacher and thoroughly familiar with the course 
and the practice tests. The students were told that there 
would be no regular class meetings. They could come into 
the laboratory whenever they wished, work with others 
or not, consult when they wished with the assistant. Each 
could progress at his own rate, test his understanding of 
each topic as he finished work on it, easily look up any 
item on which he had trouble (using the page references 
after each question), and then check himself again with 
a different punchboard test. When each student felt 
teady for a midterm or the final, he could ask for it. Each 
could thus finish the course at his own rate; all finished 
by the middle of the quarter. Results are shown in the 
last row of the table above. Over two-thirds of these 
superior and mature students made A or B grades. Some 
used the time saved in part-time employment, others in 
taking extra courses or yet other ways. Several reported 
gains in effectiveness in independent work. 

In short, self-instructional tests made it possible for 
superior students to work largely or even almost entirely 
independently in various types of special groups. Much 
time was thus saved both instructors and students, and 


students’ programs either enriched or completed sooner 
than otherwise. 


different subjects. The University of Buffalo (Mills, 1936) 
pioneered in trial of intensive Preparation for proficiency exami- 
nations. All the above studies reported favorable outcomes. 
The distinctive feature of the work at Ohio State University was 
the systematic use of the self-scoring instructional tests. 


S. L. Pressey 145 


D. MAJOR CONCLUSIONS 


The preceding pages have reported a variety of proj- 
ects. What generalizations from them seem most impor- 
tant? Four major conclusions appear to stand out from 
the total investigation. 

1. It has demonstrated a simple way to telescope into 
one single simultaneous process the taking of a test, the 
scoring of it, the informing of students as to their errors, 
and their guidance to the finding of the right answers. As 
stressed at the beginning of this paper, the usual test 
must be scored after the class meeting when it is taken. 
If the students are to know their scores and what their 
errors were, the errors must be marked on the test blank, 
and marked sheet and test questions returned to the class. 
If they are to correct their errors, they must find out 
what the right answers are. All these steps can be elimi- 
nated at one sweep, by self-scoring devices. Not only is 
time and trouble saved. Since usual test scores are not 
available until a later day, any use of those scores must 
wait till then. Self-scoring devices make possible immedi- 
ate action. Especially in any situation where expeditious- 
ness is highly important (as in a mass training program 
in a national emergency) such saving of time and trou- 
ble, and elimination of delays in use, could cumulatively 
be of decided importance. Irrespective of any self-instruc- 
tional values, simple self-scoring devices should thus be 
worthwhile as a means for saving time and labor, and 
speeding up the total testing process. 

p zA The Livestation has San that such a telescoped 
testing process, which informs each student immediately 
as he answers each question whether his answer is correct, 
and guides him to the right answer when he is wrong, 
does indeed transform test-taking into a form of system- 
atically directed self-instruction. Immediate outcomes of 
such testing with the punchboard appeared amply to 
demonstrate that this was so. Repetition of a self-instruc- 
tional test brought marked reduction in number of errors 
made. These effects showed also on tests of different 
types than the instructional tests. The learning was pri- 
marily of the subject matter and not of the test key. 
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However, use of the self-scoring device in a college course 
as a somewhat incidental feature in occasional brief tests, 
brought only limited gains in scores on later objective 
course examinations; and these gains were confined 
largely to specific items covered in the practice tests. As 
with other teaching aids, self-instructional tests must be 
made a carefully planned part of the total instructional 
program, if their values are to be adequately realized. 
Their occasional incidental use must not be expected to 
show their real potentialities. 

3. The investigation has shown that when the self- 
instructional tests were used systematically in college 
Courses as an integral part of the-teaching method, gains 
were substantial, and sufficiently generalized to improve 
understanding of a topic as a whole—eyen help on related 
topics. Thus used, the “punchboard” tests brought better 
work in regular classes, as shown by higher scores on mid- 
term tests and final examinations (even though used in 
only about a third of the course) than made by com 
parable sections of the same course not employing the 
punchboard. The device was found especially valuable 
with superior students. Sections of such students, using a 
series of self-instructional tests systematically covering a 
Course, did superior work with saving of 60 per cent or 
more in number of class hours. There was thus marked 
reduction in amount of instruction needed, and the stu- 
dents saved sufficient time so that they could take extra 
work, many finishing their education sooner in conse- 
quence. 

It appeared that substantial use of carefully planned 
comprehensive series of instructional tests could have ad- 
vantages, over the same time in discussion, somewhat 
comparable to the advantages of an objective test over an 
essay type examination or an oral quiz. More questions 
could be asked, in a given time, of every student, cover- 
ing a topic more systematically and adequately. The right 
answers in the tests could be very carefully phrased to be 
adequate. The wrong answers could bring-up and then 
dispose of common misconceptions. Such helps as page 
references after each question could guide review. The 
self-scoring device brings all these values immediately and 
incisively. Both the experimental findings and the very 
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favorable attitude of students suggest that all this was so. 

In school or college, such materials should be of value 
as instructional aids in regular classes, and of special help 
for special groups. In the Services, particularly in a na- 
tional emergency, the immediate scoring plus self-instruc- 
tion should accelerate training programs. Special groups 
in officer training might cover more subjects or complete 
training faster. New training materials sent out to ships 
or to distant ports might be distinctly more effective, 
if accompanied by self-instructional tests. Programs such 
as fostered by the United States Armed Forces Institute 
might gain in both interest and effectiveness, if such tests 
were included. 

4. This paper reports results with the punchboard. It 
was found simple and convenient even for student self- 
use, and could readily be made yet more compact and 
convenient. However, the total project has shown that 
there are various promising means for automatic scoring 
and self-instruction. A “chemocard” was tried. It was 
three by five inches and looked much like the face of 
the punchboard. Across the top was space for the stu- 
dent’s name, the date, and like information. Then fol- 
lowed two columns of 15 rows of four circles, the rows 
being numbered from 1 to 30. It could thus be used with 
30-question, four-choice tests. Each student was given 
a cheap fountain pen filled with a special red ink with 
which he wrote his name and other information at the 
top. He then marked through the first circle of the first 
row if he considered the first alternative for the first ques- 
tion to be right. If it was, the mark immediately turned 
gray (as a result of an invisible chemical printed in the 
“right” circle); if the mark did not turn, the student 
tried another circle in the first row, continuing until the 
tight answer was found. Total errors was the total of 
unchanged red marks. The chemo-card obviously operates 
to test and teach in the same way as the punchboard, but 
the materials are simpler. 


* Tt is belitved possible to print right circles with an invisible 
chemical such that any pen and ink can be used and the right 
answer will at once blur as in soft blotting paper. The idea 
of an invisible chemical to indicate correctness of response was 
developed by the writer’s former student, the late Dr. Hans 
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A key machine developed on the basis of earlier work 
(Pressey, 1926) not only indicates errors and requires 
finding of rights before continuing, but also keeps a 
cumulative count of errors and can readily be used for 
tepeated runs through the test to reach mastery. Yet 
another mechanism, again based on earlier work (Pressey, 
1927), adds to the above features in that on successive 
runs through a test, only those questions are returned 
for further drill on which a mistake was made the pret 
ous time through. Such selective review according to eac! 
student’s need should be yet more effective.12 a 

In short, “human engineering” can aid educationa 
and training programs by test-teach devices of various 
types. The major purpose of this project has been to a 
dence the value of the basic idea, as illustrated by the 
punchboard, and determine ways of using such a aa 
so as substantially to improve instruction or training. 1 
value of such devices, and the need for carefully plane 
methods for their use, if those values are to be realized, 
both seem clear. Research aiming still more to set © 
these values, and to appraise certain of the other device 
mentioned above, is now under way. 
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Knowledge of Results in 
Self-Teaching Spelling* 


J. Wuram Moore ann Wenpett I. SMITH 
Bucknell University 


Three experiments utilizing auto-instructional mate- 
tials were undertaken to obtain additional evidence on 
the use of self-teaching materials. The first two studies 
to be reported dea] with the assumption that information 
on the correctness of each response must be provided to 
the student with a minimal delay (Lumsdaine, 1960). k: 

In previous studies, immediate “knowledge of results 
(KR) has been provided in a number of ways. In some 
cases, S is informed of a correct response by some me- 
chanical or electrical device, such as a buzzer or a light. 
In other cases, particularly when the response required 
of S is based on direct recall rather than recognition, the 
desired answer is given to provide S with a standard 
against which to compare his response. In this latter 
type, both self-teaching textbooks and teaching machines 
have been used as the media for the presentation of self- 
instructional materials. 

It is claimed that by utilizing self-teaching materials, 
a student may proceed at his own tate, and that he is in 
no way restricted by the rate at which his peers learn the 
same material. As a corollary it is claimed that most stu- 


* This work was supported, in Part, by the Susquehanna 
Valley Program, Bucknell University, throuch a pant From the 
Ford Foundation. This article is reprinted from Psychological 
Reports, 1961, 9, 717-726, and it is used here with the permis- 
sion of the authors and the editor. 
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dents can learn a greater amount when using auto- 
instructional materials ad libitum than they can in “lock- 
step” instruction (Blyth, 1960). The question explored 
by the third study to be reported is, how is the rate of 
presentation of materials related to the acquisition and 
retention of the information presented? 


EXPERIMENT I 


Although there is some disagreement on the best 
method to use for providing S with knowledge of the 
accuracy of his response, most programmers accept the 
assumption that knowledge of the correct response with 
a minimum of delay is necessary for effective learning to 
take place (Skinner, 1958). A test of this assumption is 
made in this experiment. 


Procedure: Two classes of sixth-grade students (N = 
62) participated in the study. The classes, which met 
in separate buildings, contained 28 and 34 Ss respectively. 
The range of I.Q.’s (Otis Form A) in the group was 
67-123, with a mean of 100.42. To reduce the effects of 
pupil and teacher variability Ss were randomly assigned to 
experimental and control groups within each classroom. 
As a result of random assignment there were 15 girls and 
18 boys in the experimental group, and 17 girls and 12 
boys in the control group. 

Fifteen units of programmed spelling were provided 
for each S during the last fifteen weeks of the 1959-60 
school year. To test the effects of providing for S the 
correct responses with a relatively short delay each mem- 
ber of the experimental group used, in addition to the 
programmed units, a small answer booklet from which 
he could obtain the correct response to a frame within 
a few seconds after writing his response. The control 
group used the same programmed units without the an- 
swer booklet. 

The teachers? were instructed to provide each S with 


1The self-teaching materials were developed by Douglas 
Porter, Harvard University. 

2Mr. Guy Long and Mr. Bernard Zabarowski, instructors in 
the Danville Area schools, were the teachers who participated 
in the experiment. 


152 RESULTS IN SELF-TEACHING SPELLING 


a programmed unit on Monday morning during the regu- 
larly scheduled spelling period. The experimental group 
was given the answer booklets containing the correct re- 
sponses and they were told to check the accuracy of their 
answer immediately after it had been written. Studena 
were allowed 15 minutes to work on the unit; at the en 
of this time the units were collected. The same pro- 
cedure was followed on Tuesday. No other instruction in 
spelling was given. On Wednesday of each week Ss were 
given an orally administered test on the 17 words A, 
cluded in the unit. All Ss were given the same word lis 
used in the Wednesday test to study on Thursday. Op 
Friday, they were tested again on this word list. T 
procedure was followed throughout the 15 week perio A 
ll Ss were pre-tested and post-tested with the Metro 
politan Spelling Achievement Test, Form R within 4 
week prior to and following the experimental period., z 

Results: It was hypothesized that providing immedist 
Knowledge of the correct Tesponse might increase t 
amount retained over a period of weeks. If this vied 
true, it would be evidenced in a gain in spelling achieve 
ment between pre- and post-testing. Differences betas 
the groups on the Wednesday test might be attributab 
to the use or non-use of the answer booklets. Differences 
in gains between the Wednesday and Friday test scores 
might be attributable to something akin to reminiscence, 
or it may be that mode of Tesponse interacted with the 
Thursday review of the word list. 

Mean gains between pre- and Post-testing were com- 
pared for the experimental and the control groups within 
each classroom and for two classrooms combined. The 
results are shown in Table F 

It can be observed from Table 1 that the control 
groups were consistently higher in spelling achievement 


than the experimental groups; however, in no case was 
the difference significant. 


In comparing the means for the 
the two control groups and the c 
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TABLE 1, EXPERIMENT I 
Comparison of gains on metropolitan spelling achievement test. 


Mean Gain | Mean Gain 
School Control Exp. L= 
(N = 29) (N = 33) 


A 2.00 1.43 37 

B 5.57 3.35 1.18 
Combined groups 

(A +B) 3.79 2.48 1.06 


experimentals nor the controls were consistently higher 
in mean achievement score on each of the units. 

In comparing the mean gains on each unit for the 
experimental and the control groups between Wednesday 
and Friday scores, significant differences were obtained 
in only one instance. In this case, the experimental group 
was significantly higher than the control group. A com- 
parison of the mean gains between Wednesday and 
Friday test scores on the’ seventeen units combined 
again resulted in a t which failed to reach the .05 level 
of significance, 

It could be concluded from this study that providing 
Ss with KR from an answer booklet (the independent 
variable) did not aid the learning of spelling. However, 
a number of factors which may have reduced the meas- 
urable effects of the independent variable are worthy of 
consideration. First, the mean number of words spelled 
correctly on the Wednesday tests was slightly more than 
15 in both groups. When this is compared with the 
mean number of words (17) contained in each spelling 
unit, it is apparent that there were not enough words 
per unit and/or the level of difficulty of the words was 
not great enough to discriminate between the slow and 
tapid learners. 

The failure to attain a significant difference when using 
the Metropolitan Spelling Achievement Test as a meas- 
ure of retention may have been the result of the follow- 
ing: (1) all children were given the word list to study on 
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; ; ble 
Thursday of each week, possibly reducing the measura 
dieten in the post-tests; and (2), although the Mea 
politan Spelling Achievement Test may have some a 
ity as a criterion measure in this experiment, a Sa have 
posed of words taken directly from the units woul 

eater validity. f 

Pree the PaE units were used in a ke form 
cheating may have been a problem. Specifically, Be: 
probable that in a few cases Ss checked the then 
response before writing the response to the item, ediate 
teducing the hypothesized effectiveness of the imm 
KR. 


EXPERIMENT II 


: : ivocal, 

Since the results of the first experiment were E 
a second study, in which more adequate controls 
provided, was undertaken. 


Procedure: A class of 28 sixth-grade students wae 
chosen as Ss for the experiment. To approximate this 
Population of Experiment I as nearly as possible, 
study was initiated one 
classtooms with one of t 


experiment. The Ss_attending the school were from He 
same geographic area as were the students of the pt 


group. The chronological 
> to 14 years 3 months, 

12 years 1 month. The I.Q.’s (Otis 
Form B) for the group ranged from 81-128 with a mean 


A spelling test consisting of the 400 words from the 
original spelling list was administ 


ered to the Ss approxi- 
mately three months before the experiment began From 
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this test the 210 words missed most frequently were 
selected for inclusion in a revised form of the programmed 
units. To increase the level of difficulty of the task 
further, the number of words included in each unit was 
doubled. In the units used in Experiment I the number 
of words per unit was 17; in the revised units, the 
number of words varied from 32 to 39 per unit. A: a 
Tesult, six new units were formed, four of which were 
used in the four week experiment. 

A pre-test, composed of a random selection of fifty 
words from the revised programmed units was given to 
the Ss one day before the beginning of the experimental 
period. 

The programmed units were presented to Ss in a 
Foringer Teaching Machine (write-in type) to eliminate 
copying of answers. For the experimental group, the 
correct answers appeared in the window of the machine 
when the lever was pulled. The control group used the 
same machines and programmed units but did not 
receive knowledge of the correct answer. This was 
accomplished by placing masking tape over the window 
of the machine where the correct answer normally ap- 
peared. 

The actual classroom procedure was as follows: On 
Monday and Tuesday of the first week, the experimental 
group worked with the machines. On Wednesday and 
Thursday, the masking tape was placed on the machines 
and the control group completed the unit. This procedure 
was necessary since only fifteen machines were available. 
While one group worked on the programmed material, 
the other group worked quietly on other subject matter. 
A maximum of twenty minutes for completing the ma- 
terials was allowed on each of the two days. This pro- 
vided sufficient time for the slowest Ss; for the brighter Ss, 
it was more time than was required. Upon completion 
of the unit, if the specified time had not been used com- 
pletely, the student was instructed to work on assign- 
ments in other subjects. For the second week the proce- 
dure was reversed, i.e., the control group completed the 
unit on Monday and Tuesday and the experimental group 
completed the unit on Wednesday and Thursday. The 
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procedure of alternating the machines was continued 
throughout the four weeks of the experimental period. No 
other instructions were given to either group except those 
necessary for proper operation of the teaching machines. 

On Friday of each week all Ss were given a spelling 
test consisting of the words contained in the unit as- 
signed for the week. For each test, the teacher ae 
nounced each word, used it in a sentence, and pronounce 
it again. A 

On the Monday following the four-week experimental 
Period, a post-test was given to both groups. It was com- 
posed of fifty words selected randomly from the pro- 
grammed units. d 

Results: Means for each weekly test, the post-test, ano 
combined weekly tests were compared for the expen- 


mental and the control groups. The results are shown in 
Table 1, 


TABLE 1, EXPERIMENT I 
Comparison of mean scores on unit achievement measures 


in spelling. 
a ee er 
Exp. Control 
Number Group Group 
Test of Mean No. | Mean No. t 
Words Correct Correct 
(N =14) | (N =14) 
Peer een | MAN E 9 St ee 
Test 1 32 words 28.0 29.78 1.19 
Test 2 37 words 32.214 35.0 1.217 
-Test 3 33 words 27.286 30.357 1.51 
Test 4 39 words 30.214 34.286 1.367 
Post-test | 50 words 41.286 43.786 0.727 


owever, in no instance 
nt, 


Eee 
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DISCUSSION 


Since the results from neither experiment demonstrated 
significant differences between experimental and control 
groups, it may be concluded that providing S with knowl- 
edge of the correct response does not facilitate his learn- 
ing of spelling. 

Given a literal interpretation, it is possible to conclude 
that the effectiveness of self-instructional materials in 
spelling (Porter, 1959) may be attributable to the format 
of the material rather than to the use of a technique for 
providing immediate KR. This is not to imply that some 
KR was not available (and a necessary condition for 
learning) for the control group. It must be remembered 
that although the control group did not receive im- 
mediate KR by seeing the correct answer after they had 
responded, the words were used in later items. They 
could, therefore, obtain KR, though the delay in rein- 
forcement was longer. 

It is of some interest to note that in both experiments 
the means on the criterion measures are consistently 
higher (though not significantly) for the groups which 
did not have immediate KR. It is possible that seeing 
the correct response to an item before moving to the 
next item in which a similar response is required, might 
be a case of over-prompting. That is, the hypothesized 
beneficial aspects of KR in the form used in this program, 
may in fact reduce learning efficiency when the appeat- 
ance of the correct answer serves as a cue to the next 
item. Specifically, over-cueing may reduce the active 
participation of the student, thereby reducing learning 
efficiency. 

Although the data do not provide direct support for 
the use of immediate knowledge of the correct response, 
a number of desirable attributes of programmed material 
were observed in the experiment. First, both the experi- 
mental and control groups learned spelling to a high 
level of achievement without the assistance of the class- 
room teacher. In the second experiment where pre-test 


2 Some support for the disadvantage of over-cueing or prompt- 
ing is given by Goldbeck (1960). 
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and post-test data were available, the mean number of 
students who got each word correct was computed for 
each test. The mean and S.D. for the pre-test was X = 
15.33, standard deviation = 4.22, and the mean and S.D. 
for the post-test was X= 23.06, standard deviation = 
2.649. Second, the same number of words (as indicated 
in the first experiment) were learned by the students as 
in a traditional method to the same criterion level, with 
a saving of at least fifty per cent of time for all students. 
The bright students had even greater savings. Third, the 
teacher was released almost entirely from the respon- 
sibilities of dealing with the formal teaching of spelling, 
thereby allowing more time for other instructional activi- 
ties. 

In summary, these experiments support some of the 
claims which have been made for programmed materials 
in a classroom, but they question the need for providing 
KR as operationally defined in this experiment. It is 
suggested that the advantage of KR may be nothing more 
than an additional cue for the response to the next item. 
If this is the case, and the conventional KR method is 
to be used, then the number of cues within the items 
must be reduced; otherwise the response may become 
bo with little learning taking place (Goldbeck, 

On the other hand, the intrinsic value of KR may be of 
teal worth as most authorities in the field of automated 
instruction have Suggested. If this is true, then more 
research must be conducted to differentiate between the 


es and the prompting properties 


EXPERIMENT III 


__ There is little evidence resulting from current research 
In automated instruction concerning the rate of presenta- 
tion of material to the student as it efferts acquisi- 
tion and retention of the material presented.t How- 


‘Coulson and Silberman (1959) made some attempts to 


measure retention but did not relate it to the rate of presenta- 
tion of materials. 
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ever, the results of studies (e.g. Kimble, 1949; Hull, 
1943) investigating spaced vs. massed practice would 
seem to offer some evidence in favor of a controlled 
presentation of auto-instructional materials. A generaliza- 
tion of this type can be made only on the assumption that 
the use of programmed materials ad libitum and massed 
learning have in common some of the same elements 
which effect learning. 

The third study was done to compare the effects of 
completing limited and unlimited amounts of auto-in- 
structional materials, in a specified period of time, on the 
acquisition and retention of spelling by sixth-grade stu- 
dents. 


Procedure: A class of 35 sixth-grade students was 
chosen to be Ss in the experiment. The Ss were ran- 
domly assigned to experimental and control groups. 
There were 10 males and 8 females in the control group 
and 10 males and 7 females in the experimental group. 
The I.Q.’s ranged from 78 to 131 with a mean of 110.52. 
The mean I.Q.’s of the two groups were compared by a 
t-test analysis and did not differ significantly. 

To compare the effects of (a) completing programmed 
units as rapidly as possible and (b) limiting the number 
of programmed units which could be completed in a 
designated period of time, the following procedure was 
used: Six units of spelling were programmed into a self- 
teaching textbook form. The method for selecting the 
words and developing the units was described in Ex- 
periment II. The same materials were used by the experi- 
mental and the control groups. Both groups also were 
provided with answer booklets. 

The control group completed one unit each week. They 
were allowed twenty minutes each Tuesday and Thursday 
for the completion of the unit. If S did not require 
the entire forty minutes to complete thé unit, he devoted 
the remainder of his time to other subjects. On Friday 
Ss were given fifteen minutes to complete a self-admin- 
istering speiling test. The test was made up of sentences 
using the words they had used in the unit during the 
week. Upon completion of the test, the date on which the 
test was taken was recorded on the top of their paper. 
This procedure was used over a six-week period. 
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To measure the retention of spelling in the control 
group, the test which was given at the end of each unit 
was readministered six weeks later. A post-test consisting 
of a random selection of words which had appeared in all 
six units was given nine wecks after the last unit was 
completed. 

The Ss in the experimental group were permitted to 
complete the programmed units ad libitum. They were 
given fifty-five minutes a week to work on units or to 
take tests on the units. The major difference between 
the groups was that an S in the experimental group com- 
pleted as many units as possible in the allotted time 
while the Ss in the control group were restricted to one 
unit per week. If an S in the experimental group com- 
pieted a unit in one period, he was permitted to take 
the self-administering test at the end of the period if 
time was available; if not, he took it at the beginning of 
the next period. He could then proceed to the next unit. 
This procedure made it possible for some Ss to complete 
two or more units and tests during each week. When 
each unit test was completed by an S in the experimental 
group, he recorded the date at the top of the paper. To 
measure retention, the same test for each unit was tead- 
ministered exactly six weeks after each of the first tests. 
The same post-test which was adnrinistered to the control 
group was given to the experimental group exactly nine 
weeks after the mean time it took the group to complete 
all six units. 

Results: The mean achievement scores for the spelling 
test administered immediately after the units had been 
completed for the experimental and control groups were 
compared by the t-test analysis. The means of the groups 
and the results of the statistical comparisons are given 
in Table 1. 

In no case did the difference between means approach 
the required level of significance. 

A t-test analysis was used to compare the loss in 
achievement (as a measure of tetention) between test I 
and test II of each unit and the difference between the 
Scores made on the post-test by the experimental and 
control groups. Table 2 presents the data. 
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TABLE Í, EXPERIMENT NI 
Comparison of mean scores on achievement measure in spelling. 


Unit I Exp. Mean | Control Mean t 
I 28.94 28.47 38 
sag 28.39 27.29 85 
mm 34.26 34.06 17 
IV 31,06 30.75 95 
y 34.22 34.81 .39 
VI 36.82 35.38 .83 


TABLE 2, EXPERIMENT IM 
Comparison of losses in achievement on unit spelling tests. 


i _Exp._ Control 

Unit wey: X-X, : 
I 1.24 2.00 0.25 
IL 1.14 2.77 0.32 
II 1.88 - 1.33 0.15 
IV 0.53 1.80 1.46 
v 0.94 2.25 1.22 
1.19 1.81 0.18 

DISCUSSION 


The data from this study do not provide any evidence 
that massed practice with programmed materials affects 
retention adversely. The failure to obtain differences 
may be a function of the kind of programmed material 
used in this study, or.to the relatively short lapse of time 
between criterion measures used. This study lends some 
support to those who recommend that the student pro- 
ceed at his own rate when using programmed materials. 

One advantage of presenting material to Ss ad libitum 
was the saving of time. The experimental group spent a 
mean working time of only fourteen school days on the 
six units, while the control group spent the entire eight- 
een days. Further, the group receiving material ad 
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libitum appeared to be more highly motivated than the 
control group. This supports Skinners views on the 
motivational properties of automated instruction (Skin- 
ner, 1958). 
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“Actually, the two groups spent approximately the same 
amount of time on the units, for the control group, after 
finishing a unit early, worked on other materials. However, 
under conventional teaching conditions the class is kept to- 
gether, regulated by the “average student” in the class. 
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Teaching Machines: An Investigation 
of Constructed Versus Multiple- 
Choice Methods of Response* 


Epwarp B. Fry 
Loyola University, Los Angeles 


Teaching machines are defined as automatic or par- 
tially automatic devices which present a question or other 
stimulus to a student, provide a means of response, and 
then inform him of the correctness of his response im- 
mediately after he has responded. 

An exhaustive survey of the related literature revealed 
that Pressey issued the first published report of a teaching 
machine in 1926. His was a simple mechanical device 
making use of multiple-choice questions. Later he and 
his students developed studies using punchboards that 
gave immediate knowledge of results to multiple-choice 
questions. Pressey believed the use of teaching machines 
to be in harmony with such learning theory as that pro- 
posed by Thorndike. 

In 1954 Skinner proposed that teaching machines were 
effective and in harmony with such major learning prin- 
ciples as “successive approximations” and an adequate 
schedule of reinforcement (frequent rewards after small 
units of work). Skinner and his followers expressed strong 


* This article is reprinted from Automated Teaching Bulle- 
tin, 1959, 1, 11-12, and is used here with the permission of 
the author and the editor. ; 
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preference for devices which required that the student 
respond by constructing his own answer, for example, by 
writing the answer from memory rather than by recog- 
nizing and selecting one of several proffered choices. 

The purpose of this study was to determine, if possible, 
which of the two modes of response was the more efficient 
for teaching a list of Spanish words and phrases to 153 
ninth grade beginning Spanish students. A pilot study 
showed that when students were required to construct 
responses, better scores resulted on a post test but that 
a longer training time was taken. 

The major portion of the study was conducted under 
several controlled conditions in which the ‘variables of 
time and number of repetitions could be studied. Under 
Condition I, both experimental groups (multiple-choice 
versus constructed response) worked to a mastery criterion 
of two correct responses while working time was re- 
corded. Under Condition II, both groups were stopped 
before subject mastery was reached, thus allowing them 
equal total working time. Under Condition III, the 
stimulus items (English words) were presented to both 
groups by means of a large flashcard, thus allowing equal 
stimulus presentation time and equal responding time 
for each item besides controlling the number of responses 
made by each student. Learning was measured during 
two post-tests, both consisting of equal numbers of 
multiple-choice and constructed items. One post-test was 
given immediately after training, the other after a two- 
day interval. 

Results: Responses to the multiple-choice items all 
approximated the maximum possible score and hence 
did not reflect any significant differences. Constructed 
Tesponses, on the other hand, showed significant results 
favoring the constructed mode of training response under 
all three conditions for both post-tests. Under Condition 
I, where time was allowed to vary, constructed responses 
took significantly longer. 

Conclusions: Given the conditions prevailing in this 
study, constructed training responses result in more learn- 
ing than do multiple-choice responses if the criterion of 
learning is recall. The literature has indicated that the 
use of teaching machines offers great promise in the 
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teaching of many phases of nearly every school subject. 
Teaching machines enable one to make interesting ap- 
plication of learning theory and the results of learning 
experiments, and would appear to merit considerable 
attention from educators. They are excellent research tools 
for curriculum construction and psychological experimen- 
tation. 

Where it is required that the student be able to recall 
the learning material unaided, and given the conditions 
of this experiment, constructed training responses should 
be used, even if training time must be limited. 


An Analysis of Response Mode and 
Feedback Factors in Automated 
Instruction* 


ROBERT A. GOLDBECK AND LESLIE J. Brices 


American Institute for Research 


INTRODUCTION 


This paper discusses a variety of issues arising in the 
psychology of learning as applied in automated instruction. 
As in earlier conventional learning tasks employed in 
experimental psychology, discussion of continuous dis- 
course programs of the kind pioneered by Skinner (1958) 
conveniently can center around factors relating to (a) 
stimulus conditions, (b) response characteristics, ‘and 
(c) knowledge of results, or feedback. Temporal and 
sequence factors involved in variations in these three sets 
of conditions become a focal point for much of the dis- 
cussion of learning problems and issues. However, added 
tichness in variation of conditions and sequences of 
events appears to characterize continuous discourse learn- 
Ing, as compared, for example, to paired-associate rote 
learning. This added richness brings increased complexity 
to attempted functional analysis of the learning process. 
And due to the nature of continuous discourse material 
and current methods of Programming the learning, more 


* This article is reprinted from Technical Report No. 2 of 
the American Institute of Research, and it is reprinted here 
with the permission of the author and the Institute. 
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complex sequences of stimulus, response, and feedback 
events characterize the learner’s activity for a single item 
or frame. Thus, the analysis of such learning does not 
test complete with mere recognition of a written single 
stimulus term, an overt response, and a simple feedback 
signal, in that order. $ 

Past conceptualizations of paired-associate learning, 
though extremely complex in the theoretical or inter- 
pretative aspect of the learning, in functional analysis 
tended to consist of rather simple recognition of an 
S-R-Reinforcement sequence for each pair of terms. 
Since the concept of stimulus supports, now prominent 
in continuous discourse material, was not present in 
conventional paired-associate material, and since the brief 
stimulus term was constant throughout the trials, overt 
responses and reinforcement of them figured as the promi- 
nent factors in learning theory. Also, as the exposure 
of both stimulus and reinforcement terms was rigidly 
paced, the subject was presumed to be forced to be en- 
gaged, for each item, in reacting in turn to the S, the R, 
and the feedback term. This simple, rigid assumption, of 
course, ignores the responding processes actually taking 
place covertly in the subject, and gives an over-simplified 
appearance of orderly knowledge concerning learning as a 
function of the overt response and reinforcement. Thus, 
learning came to be carefully plotted out in terms of 
response frequency as learning trials progress. Some rich- 
ness was introduced into theory by considering interpreta- 
tions of such factors as similarity, overt intrusions, and 
other sources of variation in learning speed for single 
items, but in general, the notion arose that learning con- 
sists of making overt responses and receiving reinforce- 
ment. 

It is the view of the writers that the above conception 
of learning is by no means adequate in, understanding of 
or arranging for the learning of continuous discourse, 
conceptual material. And, even though this paper deals 
with responding and reinforcement variables, the writers 
view the stimulus conditions and the implicit and covert 
behavior of the learner as prime factors in their tentative 
conceptualization of the role of the selected learning 


components discussed herein. 
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This paper is in part a reporting of some of the 
Tationale development growing out of the first major 
experiment under the present contract. In this respect, 
many of the comments expand upon the reasons for 
conducting that experiment; other comments grow out 
of interpretations of results obtained (Goldbeck, 1960). 
In addition, this paper presents some of ‘the views of the 
writers on matters which they believe require additional 
research to achieve further clarification and understanding 
of the factors in programming for effective learning. In 
this context, much of the discussion is intended to refer 
to procedures for programming of continuus discourse 
material along the general lines of Skinner’s methods. 
However, the content of the paper is more generally 
intended to call attention to basic issues in the psychol- 
ogy of learning in order to enlist interest by others in 
further research and discussion of issues. 

Initial learning, or response acquisition is the primary 
emphasis in the paper, although some of the comments 
are relevant to retention, transfer, and use of the learning. 

The experimental study cited above, and this paper 
both represent considerable focus upon two factors in 
learning the writers consider of major importance. First, 
our emphasis upon implicit and covert responding departs 
somewhat from the more behavioral language of papers 
which place relatively greater emphasis on the overt 
performance of the subject. Second, our viewpoint on 
item difficulty and the effects of right and wrong responses 
differs from the view that learning cannot be made too 
easy. Our view is based upon our concept of issues of 
cueing and prompting, learning versus performance, im- 
plicit versus overt Tesponding, and upon empirical data. 

_Regarding terminology, it is well to note that we 
distinguish “response mode” from “response pattern.” 
By response mode, we mean the rather formal aspect of 
responding, such as multiple-choice versus constructed 
response, or overt versus covert. By responding, or response 
pattern, we refer to the total implicit and overt aspects 
of reacting, the former being not directly observed. 
Although experimental investigation of some of the learn- 
ing factors we discuss would yield data which would 
aid in inferring the nature of covert activity, it appears 
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that methods and techniques for measuring effects of 
these processes more directly are to be sought. 

It is hoped that this paper will help focus attention on 
some of the many still elusive conditions of effective 
learning, so that further experimentation and communica- 
tion may result in better understanding of and program- 
ming for effective learning. 


AUTOMATED INSTRUCTION AND EDUCATION 


The development of automated teaching techniques 
and the wide interest they have stimulated among both 
learning theorists and educators have contributed sub- 
stantially toward bridging a gap that has long existed 
between the problems of learning studied in the labora- 
tory and the problems of teaching in the classroom. This 
previous lack of a beneficial interaction of effort can be 
ascribed mainly to a difference in goals; these differing 
goals in turn prescribe a difference in both subject matter 
and method. 

The primary goals of the learning theorist have been 
concerned with developing knowledge about the relation- 
ships among the variables of learning. Subject matter 
and methods have been selected especially to best achieve 
these goals. The goals of the educator have been con- 
cerned with achieving the most effective learning of sub- 
ject matter chosen on the basis of educational needs. 
The methods selected have been those which were be- 
lieved to produce the best results, judging from classroom 
experience. 

Now that some of the results from the laboratory have 
been channelized into methods for effectively teaching 
classroom subject matter, in the form of programmed 
instruction devices, there has been established a common 
ground of interest and effort among experimentalists 
and educators. But, educators are still concerned with 
some goals that have not been encompassed by pro- 
grammed instruction techniques. These techniques are 
not intended to supplant the teacher and all existing 
instructional methods, but to serve as an effective aid 
to the educational process. It is to be hoped, however, 
that the problems of the educator will continue to moti- 
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vate the development of programming techniques that 
will free the teacher to play a more productive role in 
the educational process. The contributions of the educator 
should be of considerable value in both the advancement 
of programming and the incorporation of programmed 
instruction into the educational system. 


AVENUES FOR CONTINUED IMPROVEMENTS 
IN PROGRAMMING 


The development of improved programmed instruc- 
tion techniques can Proceed in several different ways. On 
the one hand, it is Tecognized that the student and 
teacher view the learning procedure as an integrated 
Process. Since the scientists, too, must be aware that 
the various aspects or variables involved in the learning 
Process have their impact on the learning outcome in a 


combined fashion, instructional methods may be studied 
in a holistic fas 


» eventual adoption and use of 


3 should depend on some such 
comparison of complete methods. 


Auto-instructional meth 
pared are largely based 
experiments. Modificatio, 
techniques will benefit f 


are an important aspect of 
ment may come from 
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which are found to contribute heavily to the leaming 
results or from data suggesting de-emphasis and perhaps 
elimination of those features which do not contribute to 
learning success and which may, in fact, degrade the 
learning process. Findings which lead to new concepts 
and, hence, to new versions of techniques are an even 
more important contribution that can result. 


THE FOCI OF NEEDED STUDY 


There are four major focal points of the learning proc- 
ess as it occurs with programmed instruction: 
1. Learning material presented 
2. Response required 
3. Feedback supplied 
4. More learning material presented 


There are many variations possible in the operations 
specified for the student to perform at each of these 
points, and variations in the combinations of operations. 
Specifying a particular set of operations defines the 
characteristics of the auto-instructional program to be 
used. There are currently two contrasting points of view 
concerning what the characteristics of programming 
should be. 

Skinner and his followers hold that precise control 
should be maintained over the student’s pattern of re- 
sponding so that desired responses can be shaped by a 
process of careful reinforcement in some successive ap- 
proximation sequence of learning frames. It is done in 
a fashion which avoids rote learning and provides under- 
standing by eliciting the desired stimulus-response con- 
nection under a variety of conditions. 

Pressey, Crowder, and others hold that the student’s 
pattern of responding should be stimulated to the limits 
of his capability. This concept of programmed instruction 
emphasizes the thinking or less controlled and more 
implicit responses that occur during learning. Feedback 
is provided for multiple-choice responses to reinforce 
and augnient successful learning, to redirect the study 
process, if necessary, and to correct errors. 

The many possible variations in the four focal points 
of programmed instruction and their combinations, each 
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representing a particular instructional technique, involve 
responses and feedback which are covert and uncontrolled 
to some degree, whether they lean toward the shaping 
concept or the “thinking” concept. 

A major task which the writers believe to be necessary 
is to untangle the specific variables that have been as- 
sociated with particular instructional techniques and to 
relate these variables to learning outcomes. It is believed 
that such relationships depend rather heavily on the 
covert and implicit activities of the student and that the 
delineation and manipulation of variables should be 
undertaken with a recognition that active participation 
of the student is not restricted to reading what is pre- 


sented to him and writing symbols or words in the 
spaces provided. 


RESPONSE MAKING AND FEEDBACK 


The remainder of the present discussion will be 
concerned primarily with points (2) and (3)—response 
making and feedback. But, as will be apparent throughout 
this paper, consideration of the effective use of these 
variables is closely associated with the form of the learn- 
ing material, especially in terms of the cueing or prompt- 
ing characteristics. One outstanding characteristic of 
most programmed material is its orientation to the stu- 
dent and his learning characteristics. Material has been 
written so that the student will be able to follow and 
understand each item of information as it is presented. 

His onentation has been maintained by using the 
criterion of student response as the guiding principle for 
Preparation and revision of material. This emphasis 00 
successful student response in preparing material, aside 
from the use of response making and feedback during 
learning, may be credited for a substantial portion © 
eg that programmed instruction has achieved to 

A consideration of how response making and feedback 
should be used in programmed instruction to produce the 
desired learning outcome Tequires some delineation of the 
characteristics and descriptive dimensions of response 
and feedback. These characteristics can then be examined 
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for the ways in which they may be employed to contribute 
to the learning outcome. It should be emphasized again, 
however, that the operationally defined conditions of 
responding and feedback may have implications for 
producing student behavior in addition to the activities 
explicitly required by the program. These may -be media- 
tors of the learning required or they may have such 
independent effects as keeping up effort, increasing con- 
centration, maintaining alertness, etc. For work proposed 
in 1961, one goal is development of improved methods 
for study of implicit responses. Some of this behavior can 
be observed and measured if appropriate conditions are 
employed. For example, study of eye movements of sub- 
jects should help estimate time used in sequences of 
actions such as reading, re-reading, thinking, and response 

writing. Eye movement studies have been made to un- 

derstand and improve reading, and to learn how subjects 

scan a work of art. Records of such movements could be 

useful to show scanning of stimulus materials, viewing 

feedback terms, glancing back at written answers, etc. 

With appropriate apparatus, this could be a valuable 

form of investigation under controlled or paced lessons, 

as well as for study of individual differences in a less 

structured program performance. Sub-vocal throat activity 

also might be useful to supplement other methods of 

observation and measurement. 

Certain aspects of implicit behavior may not be 
amenable to direct observational techniques and must be 
inferred from behavior which can be measured. The stu- 
dent’s own implicit responses may range far beyond the 
stimulus materials, reaching back in time to make new 
associations and forward in time, possibly in anticipation 
of the next frame, the test to follow, or in making inven- 
measured in terms of the criteria or 

lanned by the programmer. Arous- 
irable, implicit activity may be more signifi- 
g the defined criterion than are overt 


perfunctory responses. Also, 
be useful in inven 


Desired Responses: Control of 


siege H a has been approached in the 


behavior, it would appear 
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past through many different methods and techniques. The 
methods used in psychology have changed through the 
years, and they do, without doubt, vary in effectiveness. 
The methods used also have historically varied somewhat 
systematically with the nature of the material to be 
learned. The theory of learning developed has reflected 
the methods and materials used in the attempts to control 
behavior by experimental conditions. Some of the typical 
methods, and resulting emphases in theory, may be 
summarized as follows: 

l. The Confirmation Method. This method, typically 
used in memory drum experiments, presented a stimulus 
(S) and required the subject to attempt to make a re- 
sponse (R), after which the correct response term was 
exposed. No hint or cue beyond the “bare” S was given. 
On the first trial, the subject had no logical basis for a 
response, the S and R terms being arbitrarily paired, and 

meaningless” in themselves. The first trial thus serves 
as an “exposure” trial, or a special form of prompting 
trial to start the learning. On succeeding trials, many 
overt errors were made due to the length and difficulty 
of the task and the absence of “stimulus supports.” Learn- 
ing over a number of trials thus rested on frequency of 
exposure, and number of times correct responses were 
reinforced or “confirmed” and incorrect responses “‘cor- 
rected” by feedback. As some tasks required stimulus 
learning, response learning, and associative learning, the 
task was hard to master under the condition of confirma- 
tion. This procedure often has been called the “anticipa- 
tion” procedure, as the subject attempts to give the cor- 
device presents the confirmation 
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ing as making of responses without direct aid on each 
item as it is exposed, and having them reinforced. The 
prompting method appears to many a less “active” 
method of learning, and thus goes contrary to the belief 
that learning must be an effortful and unpleasant ex- 
perience in which the subject is pitted against the experi- 
menter, who has arranged dificult conditions for the 
subject to master. In a recent study, it further has been 
found that prompting in only three trials out of four 
is superior to either prompting or confirmation alone on 
every trial (Angell and Lumsdaine, 1960). Thus, prompt- 
ing for each “item” separately over a number of trials 
is superior to the more gross prompt represented by Trial 
l in a confirmation condition, and the haphazard prompt 
represented by the presentation of the response term 
when the subject has failed to “anticipate.” And, as 
Angell and Lumsdaine show, performing without the 
prompt on every fourth trial represents sufficient prompt- 
ing, but not over-prompting. This indication that too 
much prompting can be given (the task can be made too 
easy) is important. It may be noted that, in continuous 
discourse material, prompting can be reduced by not 
having a prompt term on some “review” items, or by 
using an indirect cue (weaker prompt) in place of the 
exact response term as a prompt. 
3, The Cue and Context Method. In Skinner-type 
rograms, responses are often prompted, as in Cook’s 
by presenting the response word before 
asking the subject to write it. Also, less direct forms of 
prompt words are used. These the present writers call 
“cues” (Briggs, 1960). Cues may be synonyms of the 
response term, indirect hints, or other semantic helps of 
various forms. In continuous discourse programs, in addi- 
tion to prompt words and cue words, there also is the 
total context of prose of which the cues and prompts are 
vords in the frame*ask the question 
parts. Thus, some Wo ive background information, and 
to be answered, some give background in orm lon, a 
uctimulus supports” to help the subject 
some are the “stimulus supp 2 
e total sequence of material 
construct the response. © h 
ei ulus context of the responses 
is written to change the stimulus, Sie 
in order to promote understanding Or penia oa 
The cue and context aspects of programming 0 course 


experiments, 
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are generally employed to achieve conceptual learning 
tather than rote memorization. Thus, the objective of 
the program differs from the usual paired-associate con- 
dition. The point is that “programmed” material exer- 
cises more “control” than a textbook because the cues 
and prompts are carefully arranged and overt responding 
is required. This probably serves to focus effort and atten- 
tion upon each “frame” of the program, and prevents 
inattentive reading. 

4. ihe Shaping Method. Skinner has employed the 
concept of shaping in discussion of both human and 
animal learning. In the case of an animal, a reinforce- 
ment is given for any overt response which is “in the 
tight direction,” or is only partially a step toward the 
tequired response. Reinforcement continues as the animal 
makes increasingly adequate responses. In contrast to 
Skinner’s verbal programming methods, stimulus supports 
are not normally employed for animals. The first behavior 
of the animal in the experimental box is thus considered 
emitted behavior, in the sense that no signal is given 
to stimulate the animal to move about or to seck food, 
etc. Thus, the experimenter puts the animal in a special 
environment and waits for the animal to do something 
before the training (reinforcement) begins. 

In verbal programs, there is not this same deliberate 
effort to reinforce only “partly right” responses. Direct 
signals to perform are given, and stimulus supports (cues 
and prompts) are employed to guide the behavior $O 
that each response will be right, even if it represents 
learning of only a small part of a total lesson. Whether 
the term shaping should be employed in both situations 
appears somewhat doubtful. 

5. Reading and Selective Test and Review. Normal 
study methods often consist of rather aimless reading, 
with only perfunctory effort by the student to select im- 
portant points, to repeat them in review, or to organize 
the material in outlines or notes. Pressey developed his 
testing-teaching devices to improve study by calling atten- 
tion to important points covered in the reading material 
and by presenting questions to be answered. The simple 
devices used with the questions provided confirmation- 
correction, and encouraged the student to correct errors 
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by repeated trials or by re-reading relevant portions of 
the study material. Thus, the total reading material 
formed the context and presented the background 
stimuli. The test stimuli required the student to recall 
and select for himself the information from the reading 
which is most relevant to the answering of the question. 
Questions were not usually interspersed into the reading 
material as in Skinner’s programs, thus placing more 
dependence on the student’s study efforts for effective 
use of the tests. These methods may be characterized 
as less controlled, as the reading units were large and the 
reading more or less separated from the question answer- 
ing. In a sense, however, the method required more 
“construction” or “structuring” by the learner than do 
Skinner’s programs, even though a multiple-choice test 


format was employed. 


Response Frequency: If one of the values of response 
making is to elicit active learning, it is reasonable to 
ask how much response making is necessary. Making 
overt responses may be considered as simply provid- 
ing a general set to participate actively in the learn- 
ing situation. From this point of view, perhaps only 
an occasional response is sufficient to maintain inter- 
est. It is not suggested that an experiment be conducted 
to explore the range of response frequencies in order to 
determine some optimum rate of responding. It is likely 
that the subject matter would influence whatever rate 
would be most effective. Each sub-topic introduced might 
be the best occasion for a response with some material, 
while other units might be most appropriate for different 
types of material. It is worth noting parenthetically that 
a response for each sentence would have the effect dis- 
cussed previously of causing preparation of material to 
focus on student reaction, rather than op the text writer’s 


mode of thinking. 
It is possible, of cou 
be required. Aside from th 


rse, that too many responses can 

e aversions that can result from 
ing, i i frequency 

too much Gf a good thing, increasing Tesponse 

to a point at ith little or no benefit is gained reduces 

learning efficiency due to the time required for response 

making. If feedback is provided after each response, the 
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problem of increased time is intensified. If feedback is 
provided after a group of responses, there arise the possible 
problems associated with delayed reinforcement for early 
responses, and the effect of this delay upon correctness of 
later responses. h 

For students who already are inclined toward reading 
effectively and with “good concentration,” the introduc- 
tion of response making may constitute more of a disrup- 
tion of the learning process than an inducement for active 
learning. A trade-off in effects may be involved between 
frequent response making in programs versus continuity in 
thought. Skinner has adopted the technique of showing 
“panels” which contain outlines, basic formulae or defini- 
tions, or other materials of a general nature. These 
“panels” are present to view during the regular sequence 
of program events and may mitigate the possible discon- 
tinuity caused by response making as well as afford aid 
to the student in making correct responses. They also 
lend “structure” and may thus enhance understanding 
and rate of progress through the program. 


Response Relevance: Frequency of response making 
taises the consideration of response relevancy. The re- 
sponse may be an important term which is critical to the 
learning outcome or a term which has little or no bearing 
on what is to be learned, While it would seem apparent 
that choice of the Tesponse terms to be used is a most 
critical aspect of programming, the value of response 
making per se can be evaluated by requiring such per- 
functory responses as articles, conjunctions, and other 
Toutine parts of sentences. 

Responding with perfunctory or non-crucial words core 
tainly could tend to increase the rate of progressing 
through a program, and may lend readiness to read effec- 
tively a key word to follow. In general, however, it is 
likely that the subject matter or desired learning outcome 
determines the importance of response relevancy. The 


i relevant response- 
Even if the response term does not constitute that which 
is to be learned, highly relevant responses eertain ad 
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the student in deciding what aspects of the learning ma- 
terial represent the core of what is to be learned. Even 
a signal to underline key words may have a similar effect. 


Repetition: It is important to keep rate of responding 
clearly distinguished from response repetition. If the 
response term is closely identified with the desired learn- 
ing outcome, such as in spelling technical terms, it is 
apparent that much advantage can be gained from requir 
ing frequent. repetition of the response. Requiring the 
same response to be made a number of times increases 
response strength with number of repetitions. A program 
sequence provides a convenient format for utilizing this 
principle of learning, and may avoid boredom as compared 
to repetitions in the identical context. Nevertheless, the 
nature of the criterion situation may determine whether 
repetition in the same or varied contexts will be mst 
effective. Since this can be a very powerful variable, it is 
most important that, in establishing the importance of 
other variables, such as “size of step,” there is no experi- 
mental confounding involving the response repetition 
variable. Also, it may be assumed that changing the 
context for the repeated response helps achieve transfer. 


Size of Response Segment: Size of response is another 
variable which is intimately tied to the desired learning 
outcome. Spelling, for example, leads us to consider 
smaller response units than would English compositions. 
Nevertheless, in our approach to any subject matter, we 
have the option of choosing the size of our response unit 
and the possibility of some systematic change in size of 
response during the learning procedure. We will not con- 
sider here the issue of part-whole learning. The size of 
response, however, is relevant to the occurrence and im- 
plications of making errors and to the, scope of elicited 


response pattern. y eS 
Much has been said of the value of “constructing, 
has not been calibrated experi- 


a response, but this value has n e expe 
mentally if terms of true individual-imposed organization 
of materials, brief effortful fill-ins, and perfunctory copy- 


ing responses. 


Many real-life performance requirements call for large 
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units of response, as in making a speech, building a house, 
painting a picture, etc. Perhaps size of response should 
be increased as subjects gain knowledge and proficiency 
in an area of study and performance. One learns to spell 
words one at a time, pethaps a letter at a time. Speed 
in handwriting later is developed, until spelling of words 
often drops out as conscious effort. Whether different 
learning conditions are tequired for these different skill 
levels has not yet been fully established. 


Location in the Frame: The response may be required 
at various positions in the frame—from the first word 
to the last word in the frame, This position is determined 
generally by the syntax or logic of the material. Position, 
may have other special relevance, 

A response near the end of a frame benefits from all 
cueing or prompting in the frame when read in the 
normal fashion. All material in the frame is read with 
the motivation of having to respond correctly. The “re 
inforcement” follows the response with little delay. 

„tņ Tesponse near the beginning of a frame may be made 
without use of the response aids which follow it. Or the 
following material may be read until the response 1s 
brought to threshold. That is, the response requirement 
may establish a specific set to acquire the information 
needed to make the response. The response can be made 
as soon as it becomes sufficiently cued for that student 

a device for supply close-to-threshold response ad) 

1s may result in a unique read-study sequence throug 
the frame material for a student. If material following 
the response is read or te-tead after the response is made, 
it would result in a delay in the reinforcement. 

If more than one Tesponse is required in a frame, the 
relationships of response position and of response func- 
tion become more involved. One interesting possibility 
is that multiple blanks in a frame may lead to overt 
7esponses in opening portions which become more ponon 
ful cues to the final response than these same words 
written into the frame. Thus, a successful constructed 
Tesponse may be a more direct prompt than supplying 
the word in the Program. But, if the response asked for 
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early in the frame cannot be made correctly, it may be 
better that the word were supplied in the program. 

More generally, it is possible that an initial fast reading 
of the program preceding a second trial to fill in the 
blanks would employ the whole first (reading) trial as a 
cue for the correct responses on trial 2. If this were found 
successful, reduced cueing may be usable. Another pos- 
sibility is that the subject should write the response if 
he believes he knows the answer, but “peek” at the feed- 
back term if he is uncertain. 


Remoteness of Prompts: A study now is being planned 
to determine whether moving prompts and cues to an 
early frame, but requiring a response in a later frame, 
is more effective than having the prompt present to view 
at the time the response is required. 

Having responses depend on learning material that is 
not available to the student at the time the response is 
required should motivate him to read material carefully 
and thoroughly with more than the goal only to make a 
particular correct response. This advantage is in contrast 
to shaping a particular response. 


Response and Feedback Relationships: Different com- 
binations of response conditions and feedback conditions 
may shift the emphasis from whether it is the reading of 
stimulus (learning) material, the making .of explicit 
responses, the making of implicit responses, or the receiv- 
ing of feeding-back that is most critical or important in 
producing the desired learning. A basic point here is 
that, in evaluating the effects of a response variable or a 
feedback variable, it is necessary to identify what other 
program characteristics seems to have been associated 
with this variable by programmers, and to determine 
whether association is invariant. If the aSsociated program 
characteristics are found to be tied necessarily with the 
variable under consideration, that variable must be related 
to both péiformance criteria and format criteria—with 
the latter involving variables which can, in turn, be 


telated to performance criteria. 
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Form of Response: It is quite likely, however, that we 
will find that, in many cases, there is no basis for believ- 
ing that a fixed combination of variables is universally 
optimum. Consider as an example the form of response 
required. The response may assume a wide variety of 
forms. It may be written, spoken, throught, constructed, 
selected, etc. There has been much controversy over 
whether a written constructed response is preferable to a 
response selected from a set of multiple-choice alternatives. 
These two forms of responding have been associated with 
specific characteristics of learning material presentation 
and with specific kinds of feedback that are possible. It is 
feasible, however, to consider the multiple-choice alter- 
natives as a form of cueing or prompting. Many students 
may, in fact, develop their own set of response alterna- 
tives from which to select when none are formally pro- 
vided as such. Conversely, when the multiple-choice al- 
ternatives are provided, the student may construct a 
response before Proceeding to “select” one. Also, the 
selection could be indicated by writing out one of the 
alternatives instead of by making some more simple motor 
response. To break down the apparent fixed differences 
implied by these two forms of responding even further, 
we might introduce the familiar “none of the above” as 
an alternative. 

It is contended that form of response can be studied 
as a variable under conditions which free it from many 
of the closely related issues of programming and which 
will allow a more useful evaluation of the criterion situa- 
tions for which various response forms are useful. Whereas 
arge steps and multiple-choice format have been ass0- 
ciated historically in Pressey’s procedures, while small 
steps and constructed response characterize Skinner's 


methods, there is no obvious reason why these particular 
combinations should be maintained. 


Response Difficulty: The ease with which a student is 
able to make the correct res i i 


a correct response cannot be made too easy for the sub- 
ject. A second view is that the item should be difficult, 


l 
A 
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but not so hard it will be failed. A third view sees a trade- 
off between the positive value of effort exerted on a 
difficult item and the negative effect of an error. A fourth 
view is that the experience of making errors and being 
required to correct them is an essential practical aspect 
of the learning process. For the sake of making program 
writing and program revision a simpler task, it would be 
fortunate to find either that making correct responses 
should be extremely difficult for the student, or that the 
student should have no difficulty at all in making a 
correct response. There is some evidence, however, that 
the optimum point of difficulty lies somewhere between 
these extremes (Goldbeck, 1960). A reasonable place to 
expect this point to lie is near the threshold for the 
response. But, how close to the threshold? Will it be 
found necessary to develop techniques which can bring 
different students to different precise levels for responding 
correctly? The answers to these questions depend very 
much on whether we find that the response made should 
always be the correct one, or that the occurrence of 
wrong responses is acceptable and pethaps under some 
conditions even beneficial. 

The danger of heavy cueing to give a high likelihood 
of correct response is that response making may become 
an automatic activity with none of the advantages of 
active participation. 4 << 

The danger of less cueing to give a lower likelihood 
of correct response is that frequent errors might also 
discourage active participation and that errors might 
lead to misconceptions. Corrective feedback is one im- 
portant safeguard against the latter danger. An argument 
which favors a moderate degree of response difficulty is 
that the implicit response pattern induced by a challeng- 
ing response requirement enhances the effectiveness of 
feedback information. Present methods of revising pro- 
grams by tryout succeed in correcting ifems too difficult, 
but fail to eliminate items which are too easy. Perhaps 
a systematically programmed combination of response 
difficulties above and below threshold can provide the 
advantages put forward for each point of view. 


Functions of Feedback: Much of the preceding dis- 
cussion has touched in general on the role of feedback 
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in programmed instruction. Specifically, feedback may 
serve several functions: 
1. It may provide information concerning the adequacy 
of responses made. 
It may serve as a reinforcement and reward for 
responses. 
3. It may have a motivating effect on performance. 
4. It may be used to direct the next step to be taken 
in the learning program. 


ha 


In regard to the information characteristics of feed- 
back, a correct response may be confirmed and an incon 
rect response simply may be disconfirmed, or it may be 
corrected. The confirmation of a correct response can be 
indicated by a simple signal such as a light or the wor 
“correct” which can be used for all correct responses. 
The confirmation may contain somewhat more informa- 
tion if the correct answer is used as the feedback. This 
usage provides additional exposure to the correct response 
and, when used with written responses, can provide Cor- 
fection of minor errors such as spelling mistakes. 

Crowder has adopted the practice of including an 
explanation of why the response is correct, thereby taking 
advantage of this part of the learning process to introdu 
additional leaming material or to insure that the correc 
response is fully understood. There is another way in 
which some statement concerning the correct response 
could be used to provide feedback. The statement might 
not include the correct response or a direct confirmation 
of it, but could be written in such a way that the student 
would be encouraged to reconsider the stimulus or leam- 
ing material and his Tesponse to determine that he has 
responded correctly. Such a statement may take the form 
of “If your answer means that ‘such and such’ is true, 
then you are correct.” The advantage of this type of 
technique is to promote a more active and comprehensive 
pattern of student Participation. The technique also 
should help eliminate the problem that arises when 4 
variety of synonyms constitute a Correct response or when 
a large-size response (e.g, a Sentence) is required. There 
may be some parallel between this “indirect” form of 


feedback and the practice of intermittent reinforcement 
and use of subtle cues. 
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The simple disconfirmation of an incorrect answer is 
accomplished by some signal which can be used for all 
incorrect responses. The possible advantages of the dis- 
confirmation technique are that the student is discour- 
aged from using the feedback as a substitute for a serious 
attempt to make a correct response, and that it allows 
for the possibility of requiring additional attempts to 
respond correctly without the uncontrolled direct prompt 
given by the feedback. 

The correction of an incorrect response can be ac- 
complished in much the same manner as a confirmation: 
present the correct response, explain why the response 
is wrong, or present a statement that would not be con- 
gruent with the incorrect response. As discussed earlier, 
if the program is one which is prone to result in a sub- 
stantial number of incorrect responses, the correction 
feature of feedback becomes a key point in the learning 
process. ! 

The reinforcement or reward characteristics of feedback 
have been dealt with extensively in the laboratory situa- 
tion and are basic to Skinner’s development of the shap- 
ing concept for programmed classroom instruction. The 
establishment of desired stimulus-response connections 
by a process of systematic reinforcement of intermediate 
stimulus-response connections raises many of the con- 
siderations found to be relevant in the laboratory. The 
schedule of reinforcement, the specificity of reinforce- 
ment, and the extrinsic versus intrinsic properties of 
teinforcement are among the considerations for study 
in the programmed instruction format. : 

The motivating properties of feedback constitute one 
of the most powerful ingredients for success with pro- 
grammed instruction. Finding that they are able to re- 
spond correctly consistently may be a strong motivating 
factor for many students, especially those who, in the 
past, have found learning to be a very frustrating experi- 
ence. For students who have experienced much success 
in learning and who may have found that learning situa- 
tions have*not enabled them to utilize fully their capa- 
bility for success, conditions which provide consistent 
reinforcement may tend to have a negative motivating 
effect. Regardless of such individual differences, there 
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remains the question of for how long a period positive 
teinforcement will retain a motivating effect. Studis 
aimed at enhancing the motivating effects of feedback, 
especially on a longterm basis, are needed. Perhaps ee: 
teduction in feedback as learning progresses is cont 
with another prevalent concept, namely that less gui 
ance should be given as learning progresses. “i 
The feeiihack teres may have te function of die 
or guiding the learning process in either an explicit bs 
informal manner. The feedback can include the inaa 
tion to proceed to some later frame in the sequence a 
the correct response has been given, and it can indic 
mastery of material in the intervening frames. If m a 
correct response has been given, the feedback can ing ; 
the instruction to return to some previous frame or to 
special remedial branch of the program. dent 
Even without the formal branching usage, the stu H 
can make use of the feedback to recognize the need i 
review previous frames or to adjust his rate and efor 
proceeding through the program. Perhaps some of b 
complexities in branching format can be avoided pl 
developing and providing the student with simple crite 
for altering his progress through the program, in or 


that he may make most effective use of the programme 
material. 


POINTS OF EMPHASIS FROM PROJECT 
EXPERIMENTATION 


Results from the first experiment completed „under 
this contract provide a basis for additional emphasis here 


upon some of the points which have been made earlier 
in this report. Rather 
earlier, the followin 
ment. 


The experiment, reported earlier by Goldbeck (1960)> 
found that, for a program of di 


than summarizing all points ma 5 
& items are selected for special com 


nce with i F _ Three 
levels of difficulty were achiey, item difficulty. 
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When retention test scores are plotted without regard 
for time taken on the learning trial, the overt response 
group performed poorer than the others for easy items, 
but much better than the others on items of medium - 
difficulty. But, the overt group required more time for 
the learning trial, as may be expected when written 
responses are required. 

When retention test scores are expressed as number 
of test items correct per minute of learning trial time, 
the reading group is superior for all difficulty levels of 
items. From the findings of this experiment, several 
comments may be made in the context of issues discussed 
in the present paper. 

First, we observe that the writing of a correct con- 
structed response in a program appears to demonstrate 
that some learning already has taken place, providing 
the response is not simply a copying response. The ex- 
planation for this apparent learning must lie in implicit 
activity. The writing of the response may or may not add 
strength to the retention of that which is learned, depend- 
ing apparently on the extent of the cueing or prompting 
(Goldbeck, 1960). jy 

The interaction found in the experiment between dif- 
ficulty level and mode of response, in terms of a reten- 
tion criterion, appears to be telated to concepts such as 
effort, attention, speed of reading, and distribution of 
study time spent in reading, thinking, answer forming, 
answer recording, etc. The experiment failed to find 
overt responding with feedback a necessary condition for 
effective learning, at least for some materials and condi- 
tions. Similarly, we find that the making of an error 
during the leaming trial does not always signify that the 
same (or any) error will be made in the subsequent 
tetention test. 3 4 

Apparently, the negative effect of some overt errors is 
overcome by the positive effect of some portion of the 
implicit activity preceding the recording of a wrong 
answer and by the effect of the feedback term. It is 
hypothesized that, when an item is moderately difficult, 
and effort is exerted to give the tight response, a variety 
of implicit responding results, Some in “right” lines of 
thinking and some in “wrong.” If wrong predominates 
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a 
in strength or if strength oscillates, a wrong po 
be recorded, but the resulting feedback is cmb Smale 
the subject to confirm the correctness of the tig One 
tendencies. In other cases of wrong overt E ir 
learning trial, an error may be made on t a on the 
test. A breakdown of right and wrong response aA E 
learning trial, for three difficulty levels, in r resented 
correctness on the retention test, has been p 
previously (Goldbeck, 1960). 


IMPLICATIONS FOR EDUCATIONAL PRACTICE 


A ion of 

Methods of Programming: From a E i 
many of the programmed instruction ee. indicate 
bearing on response making and feedback, i er “pack- 
that we are not ready to be content with any ieh sacs 
age” of instructional conditions, however m RE 
cess and enthusiasm it may have enjoyed. weii the 
to establish fully the relative importance of vari esetully 
variable characteristics which contribute most SIE 
to desired learning outcomes, and some of the a expect 
effects of these variables. It is also reasonable to uset 
that new variables will be identified and that more d. AS 
definitions of current variables will be develops “pro- 
our knowledge about the ingredients of successi sy this 
grammed instruction techniques grows, we can app. 7 ae 
knowledge to the improvement of existing program 
to the development of new Program techniques. 


ile 
Combining of Molar and Molecular Approaches: whil 
we may view the advan 

techniques as depending heavil 
the telationships between ize 
desired learning Outcomes, it is necessary to recogni 


that the course nf experimentation will be affected ÞY 
theoretical predilections and 


learning experimentalists differ in their views on learning 
just as educators’ beliefs about how children learn vary 
considerably. Some “goog” teachers are said to be 
successful because they “challenge” students. Others 
are demanding in a way which Provokes high student 
effort. Some lead the student th 
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hold individual and group drill until mastery is achieved. 
Many use discussions to promote transfer. 

One may see the “molecular” approach of emphasizing 
simple habit formation and the “molar” approach of em- 
phasizing broader and more generally defined units of 
behavior in both the laboratory and classroom settings. 
Whereas the leaders in education have tended to ex- 
hibit concern with the more broadly defined aspects of 
learning, the contribution to education by programmed 
instruction techniques has been based largely on the 
laboratory work with relatively simple learning habits. 
Hopefully, both educators and experimentalists, each 
with points of view varying in molarity of approach, can 
teconcile their conceptual differences under the impetus 
to increase their understanding of the new instructional 
technique and to improve its effectiveness to deal with a 
broad range of subject matter. 


Programming and Creativity: It is pertinent to point 
out that using programmed instruction techniques for 
effective learning of basic curriculum materials is not an 
assurance that these same techniques are appropriate for 
more highly “intellectual” or creative behavior. It is pos- 
sible that different forms of programmed techniques must 
be developed for the different levels of education. It is 
also possible, however, that the learning techniques can 
be quite similar and that only differences in the program- 
ming of the subject matter will be needed. In either case, 
we must consider the effects of programmed instruction 
variables on how the student learns “how to learn.” It 
should be noted parenthetically that such considerations 
also may be relevant to effective learning of basic curricu- 
Tuners is in progress to use programs to 
stimulate “creativity,” we yet do not ofully understand 
the processes and the variables which produce creative 
work. Some creative people have had no formal training, 
and have been reared in an intellectually impoverished” 
home. Although some information must be employed in 
creative work, overtraining, it is suspected, may inhibit 
creativity if overtraiming contributes to rigidity. Yet, pro- 
fessional people with much education often out-create 
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untutored persons or those with “technician” training. 
The subject matter of education in the professions differs 
from that taught to the technician, with, we suspect, 
superior results. Thus, “education” is broadening in terms 
of transfer while “skill” training channelizes or controls 
behavior. 


Some Unanswered Questions of Broad Significance: 

What are the implications of these considerations for 
“controlled” programs versus “stimulating” programs? 
Would “controlled” programs produce a generation of 
uniformity and inhibit activities which have not been 
programmed, but which may be needed to meet future 
problems? Would too much emphasis upon simple habit 
formation as a means of education inhibit new frontiers 
of thought? These questions will be asked by educators 
and the public. 
_ Individual differences are known to exist, but evidence 
is conflicting as to whether differences are increased ot 
decreased by training. If these differences are needed for 
the good of society, is there any possibility that there 
could be too much use of controlled programming? Or, 
to the contrary, would basic science be better mastered as 
a start toward new developments under more highly 
controlled education? 
It is the writers’ contention that obtaining answers to 
such general questions need not be restricted to the 
philosophical or theoretical level of inquiry. Experimental 
analysis of the variables involved in programmed instruc- 
tion, of the sort discussed in this paper, will contribute 
to the increased effectiveness of programmed instruction 
techniques. This effectiveness, in turn, must be evaluated 
in terms of educational goals. Thus, a balance between 
empirical tests of programming methods and research to 
improve programming technique should pay off in the 
long run. 

Lest the teader feel that some of the questions raised 
above contain too much of a “viewing with alarm” O 
the unknown longterm effects of controlled programming, 
we hasten to state that we advocate sufficient experience 
with the methods to help answer our own questions. Also, 
there will be certain checks and balances which accom- 
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pany trial of programs. Normal procedures will not be 
abandoned overnight. Whether due to cultural lag or to 
wisdom, the first widespread use of programming in 
education will be as another tool to help get the educa- 
tional job done. Thus, a “systems” approach, whereby 
programming is incorporated into total procedures where 
it proves most useful as a component of total methods, 
will result in a favorable balance of procedures. Even- 
tually, educational objectives must be studied systemati- 
cally in terms of methods effective in achieving them. 
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A Comparison of Several Types of 
“Immediate Reinforcement” * 


J. Wittiam Moore ann WENDELL I. Smu 
Bucknell University 


An application of reinforcement theory has received 
particular attention from those working in the field of 
automated instruction (Coulson and Silberman, 1960; 
Lawson, 1960; Lumsdaine, 1960). Specifically, much of 
the demonstrated effectiveness of auto-instructional ma- 
terials has been attributed to immediate knowledge of 
Tesults (KR) as a reinforcer. wk 

While the value of immediate reinforcement in in- 
creasing the rate of learning has been demonstrated, the 
assumption that immediate reinforcement (when the 
reinforcement consists of KR) as defined in the animal 
laboratory or in motor learning tasks is synonymous 
with immediate reinforcement through KR as used in a 
1961) learning task, may be questioned (Moore & Smith, 

Most of those who have questioned the reinforcing 
Properties of KR, have not questioned the general con- 
cept of reinforcement, but they Propose that a distinction 
be made between confirmation and reinforcement of the 
response. Carr (1959) states that “confirming the cor- 


* This preliminary report is based on a stud ted by 
the Air Force Office of Scientific Research, Sie AF- 
NOS ELSA, and is used here with the Permission of the 
authors. 
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rectness of a learner’s responses to problems may be 
expected to be reinforcing only if the learner’s motiva- 
tion is intrinsic to the task being learned.” 

This problem is further complicated (whether KR is 
classified as confirming or reinforcing) by the limited 
amount of empirical evidence available concerning the 
effect various methods of providing KR have on human 
learning. For example, KR can be provided by (a) the 
immediate knowledge that the given response is correct 
(a flashing light for a correct response), (b) immediate 
knowledge of what the correct response should be (the 
correct answer to the stimulus item appearing immedi- 
ately after the student responds to the item), (c) im- 
mediate reinforcement for correct responses using an 
extrinsic reward (a piece of candy or a coin presented 
for each correct response), and (d) KR provided by cues 
which appear in succeeding items (a normal condition in 
the linear program utilizing the concept of successive 
approximations for increasing the likelihood of making 
the correct response). In the last example, this method 
of providing KR could be in addition to, or in lieu of, 
the conventional method of providing KR in programmed 
instruction. 

From the examples cited, it can be seen that methods 
for providing KR vary greatly in their reinforcing and 
in what may be called their “guidance” properties. For 
example, one would expect a tangible reward (money) 
to have greater drive reducing properties for some people 
than would the more intangible reward of KR. 

Further, if KR, where the correct response appears im- 
mediately after the student makes his response, has only 
“guidance” properties, then this type of KR is unneces- 
sary (assuming that the linear form of programming is 
being used). This is based on the assumption that suffi- 
cient cues should be built into the successive approxima- 
tions of the items to assure a high probability of making 
the correct response. 

The purpose of this investigation was to compare the 
effects of, various types of KR in leaming psychology 
from programmed material. 


; ha 
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PROCEDURE! 


Subjects: Two sections totaling 220 university wee 
enrolled in introductory psychology for the secon ie 
of the 1960-1961 school year were randomly assigne Hee 
one of 10 groups. To reduce the effects of the teache 


parable motivation for all groups, Ss were told that a 
Part of their semester grade would be based on their 
achievement on examinations based on programmed ma- 


Experimental Period: The experimental period con- 
sisted of six weeks beginning with the second week of 


days a week during the six-week period. In no case were 
the laboratories? scheduled on two successive days. All 


students were expected to complete 96 frames during 
each laboratory, 


Experimental Groups: Since the amount of learning as 
measured by two tests of achievement could be related 
not only to the types of KR, but to the type of response 


and constructed-response items 
were established to compare 


* Our appreciation is expressed to Douglas Candland and 
ary Jane Mordan for their extensive Participation in the 
conduct of this experiment. 


oratories refer to those Periods during which students 
worked On programmed material, 


omonski and Eigen (1960) completed a stud comparing 
programmed texts and teaching machines and did not obtain 
Significant differences on the criterion measure, 
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were established. Experimental groups one (Eme), two 
(Eme), three (Esme), four (Emc) and five (Esme) 
used programmed materials with multiple-choice (mc) 
type answers. In group E,mc-E,mc the material was 
presented in a teaching machine, varying only the method 
of providing KR. In the fifth group the material was 
presented in a programmed text form with KR provided 
in the form of the correct response. Experimental groups 
one (E,cr), two (Eger), three (Egcr), four (E,cr) and 
five (E;cr) used programmed materials which required 
constructed-responses (cr). As with the multiple-choice 
groups, teaching machines were used with the first four 
constructed-response groups and a programmed text was 
used for the fifth group. The 10-celled chart shown here 
presents the experimental groups with the various condi- 
tions and types of KR. 


EXPERIMENTAL DESIGN 
Mode: Ee 
Con- 


Knowledge of Results | Multiple | structed 
Choice | Response 


|, |e ee 

No knowledge of re-| E: E Only stimulus mate- 
sults rial 

i- 


EPN eee 


Immediate knowledge E: E: 
of what the correct 
response should be 


The correct answer 
appearing after 
the response had 
been made 


mi A ee | t 


Immediate knowledge Es Es 
that the given re- 
Sponse is correct 


Stimulus materials 
plus a flashing 
light for correct 
responses 

e a o) —— 

Immediate knowledge| Ex E 
of results plus ex- 
trinsic reward 

Senei ia sa 

Immediate knowled E E: 

ge = 
of what the correct 


response should be 


> One penny for each 
correct response 


Programmed text 
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Facilities and Equipment: Two regular classrooms were 
scheduled for use by the 10 groups. Foringer write-in 
machines were used for presenting the materials to the 
constructed-response groups. The E,cr group machines 
had black masking tape covering the space on the ma- 
chine where the correct response normally appeared. This 
prevented S from knowing whether the response was cor- 
tect or not. Escr’s used the Foringer machine in the con- 
ventional fashion (the correct answer appeared immedi- 
ately after S made his response and pulled a lever). The 
Escr group used a machine equipped exactly like the 
Eyer group except that a small light was attached to the 
upper right-hand corner of the machine. This light was 
controlled by an O seated behind S and for each correct 
response which S made O flashed the light, signaling that 
the response was correct. If the light was not flashed 
immediately after S made his response, S knew that he 
was to move to the next frame. In no case did O com- 
municate orally with S. The E,cr machine-observers com- 
bination was exactly the same as the Escr; however, O 
kept a running tally of the correct responses. At the end 
of each laboratory period S was paid (by the laboratory 
proctor) one penny for each correct response. The E,cr 
group did not use the Foringer machine but used a pro- 
grammed textbook. 

The multiple-choice groups used specially designed 
three-choice-response teaching machines. The programs 
were placed in booklet form to the left of the machine. 
On the face of thè machine (between two rows of re- 
sponse buttons) an answer sheet containing three possi- 
ble responses for each frame was placed. The Ss were 
told to select a correct response, circle it on the answer 
sheet, then press the corresponding response button on 
the machine. The E,mce group used a machine which 
kept a record of the number of responses but did not 
provide KR. Esme used a machine equipped with three 
lights (keyed to the three choices on the answer sheet). 

€ machine was so constructed that regardless of 
whether the correct response button was pressed; the light 
indicating the correct response would light. The machine 
also kept count of the number of Tesponses which S 
made. E3mc used a machine which differed from the 


ee 
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E,cr machine in that it had only one light on the panel. 
When the correct response was made the light would 
light; when incorrect, it did not light. This machine was 
also equipped with counters which not only counted the 
total number of responses the Ss made, but also kept 
count of the total number of correct responses. The 
E,mc group used a machine exactly like the Eyme and 
was paid one penny for each correct response. This pay- 
ment was made at the end of each laboratory period by 
the proctor. The E;mc group used the same materials 
and followed the same procedures as the other multiple- 
choice groups. A record was made of the numbers of cor- 
rect responses and total responses for each S at the end of 
each period. 

Each laboratory had one proctor assigned for its gen- 
eral supervision (the distribution of materials and direct- 
ing of Os). An average of one O for each three Ss was 
provided in each of the laboratories. The purpose of these 
Os was to make certain that the Ss completed the pro- 
gram as stated in the experimental design. The presence ` 
of several Os helped to reduce the “observer variable” 
introduced in the Eger and E,cr groups. In no instance 


Materials: Eleven hundred fifty-two frames of the 
Holland-Skinner psychology program were used.5 This 
program was selected because of the careful manner in 
which it was developed. The constructed-response por- 
tion consisted of placing 96 frames on each of 12 rolls of 
materials for use in the Foringer machine. One roll was 
used during each laboratory session. Since the conven- 
tional Holland-Skinner program is in the constructed- 
response form, a modification of the program had to be 
made to adapt it to the multipie-choice machines. This 
modification consisted of constructing three answers for 
each item, only one of which was correct. Otherwise, the 
programs were exactly alike. £ : 3 

No other formal instruction in learning theory was 
provided for the Ss during the experimental period. 


í In these groups one O was necessary for each S. 
did Os provide information about the program to the Ss 
3 Permission to use this material was granted by James H ik 
Jand to whom we express our appreciation for this courtesy, 
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Evaluation: A 

en i 


closely related to the items contained in the progra 
At the end of the experimental period, a final ES a 
tion was administered which included all of the cone 
Presented during the experimental period. It Kee co 
up of one-half multiple-choice items and one-ha in aa 
structed-response items. This type of ermaaiou wide 
dition to being a measure of achievement cou Ena 
some information concerning the interaction rogram 
the tvpe of Tesponse element employed in a REE e 
and the type of test item used in evaluating the stude: 


3 ¢ Á er some 
Period. It was expected that this might pen by 
relationship between the achievement, as meas 


Results: Since this is a 
derived from the two crit 


means resulting from the OPA 
9 provided, the method of responding, and t Cna 
teractions of these variables. Tables 1 and 2 summa 


he effects of the various 
Second, diff 
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TABLE 1 


Analysis of variance of achievement test scores (test I). 


Source of Variation df ms F 
Mode of response 1 33.49 1.16 
Knowledge of results 4 59.99 2.09 
Interaction 4 11.73 0.41 
Within 177 28.75 

TABLE 2 


Analysis of variance of achievement test scores (test TI). 


Source of Variation df ms Ẹ 
Mode of responses 1 29.60 0.17 
Knowledge of results 4 372.11 2.12 
Interaction 4 29.45 0.17 
Within 180 175.48 


some Ss in working with the material may have reduced 
differences. For example, it was possible for those Ss who 
used multiple-choice teaching machines and programmed 
texts to refer to a previously completed frame (one of the 
functions of the Os was to see that this was not done), 
while referral to a previously completed item was not 
possible for Ss who used constructed-response machines 
(Goldbeck, 1960). 

If the criteria measures were valid, and if the failure 
to reach a satisfactory level of significance was not the 
result of uncontrolled variables, then other explanations 
are possible. For example, the comparisons made be- 
tween groups using the constructed-response machines 
with and without KR, are based on well-controlled experi- 
mental conditions. Two, or the combination of two, hy- 

otheses may account for the failure of these comparisons 
to differ significantly. First, the “no KR” groups may 
have received KR in the succeeding frames (Moore and 
Smith, 1961). If this is the case, it could be questioned 
whether this type of KR was “intrinsically teinforcing.”” 
(For KR of this type to be intrinsically reinforcing, it 


n” 
200 TYPES OF “IMMEDIATE REINFORCEMENT 


would seem logical to assume that enough informal 
would have to be obtained from the frame to provide 
absolute assurance that the preceding item was conei 
or whether the KR was sufficiently subtle that it provides 
some guidance in answering the item but still was no 
intrinsically rewarding. Second, the group which Mier. 
provided with immediate KR may have been force ra 
respond more actively to the individual stimulus aena 
than the group which had immediate KR (Gold ae 
1960). If this were the case, it could be assumed that th 


effect was sufficient to counterbalance the positive effects | 


of the intrinsically motivating properties (if, in faot, H 
is a result of KR) of immediate KR. A further fin E 
resulting from that part of the study in which better od 
trols were possible, was that the groups using the P 
grammed text and those using teaching machines did a 
differ significantly. In fact, the mean achievement ete 
on the criteria measures were consistently in favor of S 
programmed text. This finding has been reported by 
Komonski and Eigen (1960), also. oa iE 
An interesting observation was made concerning é 
tank of the mean number of errors on each of the oe 
tests for both the constructed-response and mule 
choice groups. It was noted that a smaller mean num a 
of errors was made (tank) by the groups using the pro 
grammed texts and the groups which were receiving T 
extrinsic reward than was made by the other groups. ! 
significance can be attached (because of relevant post- 
tions of mean errors on the criterion measures) to this 
observation, it Suggests that (1) the teaching machines 
are not as an effective a media (in terms of learning) for 
presenting programmed materials as is a programme 
text, and (2) an extrinsic reward (though in this case it 
might be better classified an incentive rather than a 
reinforcer), when added to KR, i 


i ` » 1S more effective 1n 
reducing error rave on a criterion measure than KR by 
itself. 


At the time of the writing of this preliminary report, 4 
complete analysis of the data has not been mude. How- 
ever, an analysis of crror rate on Programmed material is 
being done, as well as a tabulati 


questionnaire which was designed 


_— r a 


J. W. Moore ano W. I. Smiru 201 


ation of interest and attitude toward the experimental 
methods used in this investigation. Comparisons will be 
made of the types of KR and the types of responses on 
the error rate of the programmed material. Comparisons 
also will be made relating error rate on programmed ma- 
terial and scores achieved on the criteria measures. The 
questionnaires will provide some information concerning 
the relationships between interest in the programmed 
material and the level of achievement defined in terms of 
error rate and of scores on the criterion measure. 
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Size of Step and Cueing* 


WENDELL I. SMITH AND J. Witt1am Moore 
Bucknell University 


One of the common problems of a programmer who 
is using the linear form of programming is to determine 
the size-of-step between frames. The concern about the 
concepts of size-of-step and cueing is based on the desire 
to increase the probability that a student responding to 
any given item will make the correct response. The size- 
of-step may vary in at least two ways. It may vary in the 
telationship of a succeeding item to a previous item (the 
degree of change in the reduction of cues in the items), 
and, two, it may vary in the number of cues within an 
item that is not dependent on a Previous item. 

The concept of size-of-step, by definition incorporates 
the principle of cueing and it may, in addition, influence 
the rate and amount of learning through the frequency 
with which Tepetition of the correct response is required. 
On the other hand, the concept of cueing places less 
emphasis on learning through the repetition of the cor- 
tect response in a series of frames, and places more atten- 
tion on the eliciting of the Correct response within a 
given item. It is on this basis that the concepts, size-of- 
step and cueing ase differentiated in this paper. 

The importance of this Problem is eyident when atten- 


* This is a preliminary Teport of a stud ted by 
Grant No. 736087, Uni Buy SURDON 
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tion is given to the conventional methods ef program- 
ming in the linear form. Usually the size-of-step (i.e., the 
number of frames included in the program) is deter- 
mined by the slowest (lowest ability) test student used 
in the development of the program. The assumption 
usually made is that if the size-of-step is decreased to the 
point where the weakest student can complete the mate- 
tial with a minimal error rate, the optimum program for 
all students for whom the program was written will have 
been developed. 

Some evidence is available (Goldbeck, 1960) which 
indicates that a low error rate on a program does not 
necessarily mean optimum achievement on a criterion 
measure. This suggests that providing larger steps (either 
by a smaller number of repetitions in responses for a 
given concept or by reducing the cues within a frame) 
does not necessarily reduce the efficiency in learning, 
although it may increase error rate. Therefore, it is de- 
sirable that more attention be given to the performance 
on the criterion measures as a means of evaluating pro- 
grammed material than to the error rates for a program. 
This is not to imply that a relationship does not exist 
between the error rate a student makes on a program 
and the score he achieves on a criterion measure, but it 
does suggest that excessively small steps and/or strong 
cueing (to assure minimal error rates) may not produce 
the highest scores on a criterion measure. 

The purpose of this study, therefore, was to compare 
the effects of three different sizes-of-step, amount of cue- 
ing, and the use of illustrations on the spelling achieve- 
ment of fifth-grade children. 


PROCEDURE 


Subjects: Three sections of fifth-grade Ss were selected 
for participation in the program.1 In order to have a popu- 


1 Fifth-grade students were used as subjects because the 
experimental period was scheduled for the last four weeks of 
the academic year. Since the spelling materials were based on 
sixth grade spelling lists it was felt that these subjects would 
be more typically sixth graders (because it was the end of the 
yeas) therefore the material should be most appropriate for 
them. 
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lation representative of the school district? in which the 
experiment was conducted, one grade each was selected 
from three of the four elementary schools. The 96 Ss 
were then randomly assigned to one of six experimenta 
conditions. To reduce the effects of the teacher variable, 
the students: were assigned to one of six groups within 
each room, thereby creating six experimental conditions 
within each room. The mean I.Q. of the group Was 
105.58 and the mean age at the beginning of the exper: 
mental period was 11 years 1 month. As a further control, 


the students were matched on the basis of sex before 
assignment. 


„Experimental Treatment: A list of 166 words was COM 
piled by selecting words most frequently included in l 
widely used sixth-grade spellers. To compare the ¢ ects 
of size-of-step and cueing on learning, six different Pr” 
grammed forms of each unit were written.’ The forms 
„were: a small-step program (4-9 frames per word) ime 
forms, one with pictorial cues, and one without pictorial 
Cues; a medium-step program (3-7 frames per word), one 
form with pictorial cues, the other form without pictori 
cucs; and a large-step program (3-6 frames per WO" 
with the comparable forms as indicated for the first tw° 
Programs. For pictorial cues, an illustration which was 
appropriate for each word was sketched on the program 
beside the word on its first occurrence. All materials wet? 
pona in a self-teaching textbook with a separate book 
eenas the correct answers. The students Wete 
e ne complete each item, then immediately COD” 
Bede Oo neeme of their response from the answer 
materials O R general instructions for using the 
provided. All So mtance in the teaching of spelling was 
and proceed to ma instructed to complete each frame 
pane ne next one without reviewing previous 

The total i i 
during which Se Couple a er fear vee, 

© units each week. The Ss 
b 


* This study was conducted i k 
Daai Pennsylvania. The Dor RS schools, 
elementary supervisor, and his staff made the oot Hauck, 
The programming was done by Mary iene ee 


W. I. Smr anp J. W. Moore 205 


were given a maximum of 35 minutes for the completion 
of each unit.* Tuesdays and Thursdays were used by all 
three teachers for completing the units. On Friday an 
orally administered spelling test consisting of a list of all 
words included in the two units was given by the teachers 
to the groups. This procedure was used on all four weeks 
of the experimental periods. 

In addition to the weekly tests, the Ss were pre-tested 
and post-tested respectively one day before and one day 
following the experimental period. This spelling test con- 
sisted of a random selection of 50 words out of the total 
spelling list. 


RESULTS 


To obtain evidence that the various experimental 
methods were effective in teaching spelling, the mean 
gain between the pre-test and post-test for all Ss was 
computed. The mean gain was 11.23 words. Since these 
tests were made up of a representative sample the spell- 
ing list of 166 words, an estimate of the total mean gain 
in spelling achievement can be obtained by multiplying 
the mean (11.23) of the sample by 3.33. The estimated 
mean gain in achievement over the four weeks of the 
experimental period was 40.96 words. 

To evaluate the effects of the several experimental con- 
ditions on the Ss’ spelling achievement, the means on 
each of the weekly tests were compared by the use of the 
F-test. á 

In none of the four comparisons were the differences 
between means sufficiently great to reach a satisfactory 
level of significance. 


Discussion: Although only preliminary analysis of the 
data has been completed at this time, neither the size-of- 
step nor the pictorial cues seemed to dffect significantly 
the scores on the criterion measure. This would suggest 
that the range in size-of-step and cueing from the small- 


«The 35-minute periods proved to be a sufficient amount of 
time for even the children who were the slowest workers in 
the groups. Ifa student completed the unit before the maximum 
time, he was instructed to work on other subjects. 
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step-pictorially-cued program to the large-step-non-pictori- 
ally-cued program all met the conditions necessary for 
learning. Since the large-step program proved to be as 
effective in teaching spelling as the small-step program 
(and it could be considered more efficient since logically 
it would require less time to complete) it may be con- 
cluded that programs consisting of steps which are very 
small are not necessarily an efficient method of instruc- 
tion. Further, if very small steps and over-cueing are 
interpreted by S as unnecessary repetition, then the addi- 
tional hazard of loss of interest in the material may be 
an undesirable by-product. Since this is a preliminary 
report, error rate on the individual programs has not been 
computed, therefore, comparison of error rate an 
achievement on the criterion measure have not been 
made. If it is found that a greater number of errors was 
made on the largest step, unillustrated program than on 
the smallest step, illustrated program, there will be sup- 

` port for Goldbeck’s finding that a low error rate on the 
instructional materials does not always yield highest 
achievement on the criterion measure, 

It should be noted that all six forms of the program 
provided an effective method of teaching spelling without 
the assistance of a classroom teacher. | 

It is concluded from these preliminary data that Ae 
necessity for small steps and strong cueing to assure the 
eliciting of the correct response to the frame of a pro- 
gram may have been overemphasized; that larger step 
programs (an optimal level must be identified) may have 


greater motivational value, and they may teach as much 
as effectively in less time. 
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Automated Teaching and Individual 
Differences* 


Jonn E. Courson anp Harry F. SILBERMAN 
System Development Corporation 


In 1958 an automated teaching project was formed by 
members of the Research Directorate at the System De- 
velopment Corporation. The functions of this project 
were to investigate the current status of automated teach- 
ing, to conduct research on variables of automated in- 
struction, and to determine what contribution SDC, as 
a non-profit corporation, might make in the develop- 
mental and applied aspects of this field. 

Project members began with a review of the “state of 
the art” as it then existed. This review indicated that, 
despite an apparent diversity in design, most teaching 
machines had one characteristic in common: a relative 
inflexibility of operation. With a given set of instruc- 
tional materials (“items”) in the machine, each student 
working with the machine would receive the same se- 
quence of items, regardless of his individual capabilities 
or limitations. 

The basic assumption underlying the design of a fixed- 
sequence teaching machine is that a single sequence of 
items can be optimally effective for a number of stu- 
dents. This assumption seemed highly doubtful when 


— r 


* This article originally appeared in Audio Visual Com- 
munication Review, 1961, 9, 5-15, and is reprinted here with 
the permission of the authors and the publisher. 
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viewed in the context of the typical classroom, with 20 
or more students differing not only in training experience 
but in inherent ability as well. It appeared that an ideal 
sequence of items for one student would be less than 
maximally effective for some other student. 

Using certain behavior characteristics of the human 
tutor as an initial guide, project members considered 
possible teaching-machine designs which might use dif- 
ferent instructional procedures for different students, de- 
pending on the students’ performance during training. 
This adaptable feature was termed “machine Tesponsive- 
ness.” It was clearly recognized that well-written, well- 
tested instructional items were essential for any successful 
automated instruction. Without such materials any 
teaching machine would be completely useless. Given a 
good set of items, however, it was believed that the inter- 
action between student response-behavior and machine 
selection of materials might be an important factor in the 
effectiveness of automated instruction. 

In April 1959, the Automated Teaching Project con- 
ducted an experiment with manually operated teaching 
machines to explore the effectiveness of machine re- 
sponsiveness, as compared with a fixed sequence mode 
(Coulson and Silberman, 1960). Responsiveness was in- 
troduced by a simplified form of “branching” in which 
a student who correctly answered an item on a given 
topic was not shown other items on the same topic.t In 
this way each member of the “branching” group received 
a (different) subset of the items seen by the fixed se- 
quence group. Other variables investigated in the same 
experiment were student Tesponse-mode (multiple-choice 
versus constructed-response) and item step-size (many 
small steps versus fewer large steps). Materials in elemen- 
tary psychology were Presented to 80 junior-college stu- 
dents. Results of the study were as follows: 

1. Training with the manually controlled machines 
yielded significant student learning in each of the experi- 
mental groups. 


g was unquestionably 
crude, and the criterion for branching somewhat elite It 
was found, however, that more complex forms of branching 
were not feasible with the manually Operated machines. 
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2. “Branching” students required significantly less 
training time than “fixed sequence” students, and did 
not differ on a post-training criterion test. 

3. Students receiving many items with small steps 
learned more than students with fewer large-step items, 
but also required significantly greater training time. 

4. “Multiple-choice” students required less training 
time than “constructed response” students; the two 
groups did not differ on criterion performance. 

The results of the initial experiment, while not defini- 
tive, pointed toward an advantage in favor of machine 
responsiveness over the fixed sequence mode of instruc- 
tion when training time was taken into consideration. 
At the same time, experience with the manual devices 
clearly indicated the desirabilitv of more automatic equip- 
ment capable of controlling complex patterns of instruc- 
tional variables. Project members believed that a com- 
puter-based teaching machine would offer maximal re- 
sponsiveness to individual differences, as well as a high 
degree of flexibility as an experimental tool. 

Before designing a teaching machine with responsive- 
ness to individual differences, however, it was necessary 
to explore the effects of negative reinforcement in auto- 
mated teaching. 

Since a “responsive” .teaching machine uses student 
errors to determine the sequence of items, it follows that 
most students will make at least a few errors during each 
training session. If this were not true, all students would 
teceive the identical sequence. Consequently, an experi- 
ment was performed to determine the effects on learning 
of student errors and resulting negative reinforcement 
(Melaragno, 1960). Again manual equipment was used, 
this time to present instructional items in logic to 28 
junior-college students. Results indicated that a relatively 
small number of errors (negative reinforcements) dis- 
persed among correct responses did not affect student 
learning adversely, but that massed negative reinforce- 
ments did hinder learning. The conclusion drawn was 
that student errors could be used for sequence control 
in a responsive teaching machine, but that items should 
be so written that the student would not miss many 


items in a row. 
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COMPUTER-BASED TEACHING MACHINE 


In early 1960, SDC completed work on an experimen- 
tal teaching machine consisting of three major units: (1) 
a Bendix G-15 computer; (2) a random access slide pro- 
jector; and (3) an electric typewriter. 

l. Bendix G-15 Computer. This serves as the central 
control unit for the teaching machine. It contains the 
master control program which governs the operation of 
the entire machine. The computer determines, at all 
times during a training session, what materials are to be 
presented to the student. It analyzes student responses to 
the instructional material and compares these responses 
with stored data. 

Rolls of paper tape, each of which can hold the neces- 
sary information for a different instructional lesson, are 
contained in easily interchangeable magazines. A bell 
mounted in the computer frame can be tung under com- 
puter control, thus permitting programmed auditory sig- 
nals. 

2. Random Access Slide Projector. Developed by the 
engineering staff at SDC, this projector holds up to six 
hundred 35mm slides in 15 magazines of 40 slides each. 
It receives instructions from the G-15, selecting and 
projecting instructional slides in the sequence indicated 
by the computer. 

3. Electric Typewriter. Two primary functions are 
served by the typewriter, which is linked to the Bendix 
computer. First, the student uses the keyboard to insert 

1s answers to the instructional materials, normally in 
multiple-choice form. Second, the computer program 
takes control of the typewriter to print messages telling 


the student how successful he has been in answering the 
questions. 


FUNCTIONAL CHARACTERISTICS OF EXPERIMENTAL 
MACHINE 
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receives appropriate knowledge of results after each at- 
tempt. Every item presented to him is registered within 
the computer control unit as having relevance to one or 
more specific topics to be taught. The computer keeps a 
record of the student’s performance on each item and a 
cumulative record of his performance on each of the 
topics covered by the items. These performance measures 
may include the student’s response times as well as his 
error count. If student performance falls below a certain 
level for a particular topic, the student is branched to a 
special set of items which provide remedial work on that 
topic. He may be taken through the entire remedial rou- 
tine, or may be brought more quickly back to the basic 
series if he performs well on the remedial items. 

On returning from the remedial routine the student 
again receives items from the basic series, continuing 
until he has finished this series or until his performance 
once more falls outside certain programmed limits. In the 
latter case, he is again branched temporarily from the 
basic series. 

In certain cases, a single missed item on a particular 
topic may be sufficient to take the student to a remedial 
routine. The teaching machine not only recognizes that 
an item has been answered incorrectly, but also considers 
the specific incorrect alternative chosen. Certain answers 
may be indicative of a serious misunderstanding by the 
student and may cause the machine to branch immedi- 
ately to a remedial sequence. 

If a student does very poorly in his training and re- 
quires excessive remedial work (according to programmed 
criteria), the machine may take him completely out of 
the original series, branching him into an alternative basic 
series. This alternative series represents not simply a 
remedial routine, but a major change in the training 
approach. The original series might, for instance, use the 
procedure of first giving general rules, and then present- 
ing examples to illustrate the rules; the alternative series 
might reverse the order, gradually building up to a gen- 
eral rule by a progression of illustrative examples. 

In the event that a student’s performance on some 
topics of a basic series is sufficiently high, the teaching 
machine skips certain items for these topics, thereby 
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allowing the brighter student to progress rapidly to more 
advanced topics. 

At various points during the training session the stu- 
dent may be required by the machine to indicate his 
degree of confidence concerning his own learning prog- 
tess. If he expresses a feeling of confusion or lack of 
understanding, he may be taken to a remedial branch 
even though his actual performance is above the normal 
criterion level for such a branch. An expression of great 
self-confidence, on the other hand, may cause the student 
to be returned more rapidly from a remedial routine to 
the main sequence. 

The operation of the experimental teaching machine 
may be illustrated by means of the flow chart opposite 
(Fig. 1). This chart does not represent an actual instruc- 
tional sequence, but is merely used to demonstrate some 
of the types of decisions that can be made by a computer- 
based teaching machine. Each of the numbered squares 
symbolizes an instructional item. After the student has 
answered questions in items 1 and 2, the computer deter- 
mines whether he has missed either question. If he has, 
the computer gives him remedial items 3 and 4. If not, 
he goes directly to item 6 (representing a new topic). 

On item 4 the correct answer, b, takes the student 
directly to item 6. Answer a, representing a particular 
type of error by the student, causes him to receive item 3 
again. Answer c, another error, causes him to be given 
an extra remedial item, 5, before going ahead to item 6. 

Item 12 represents a self-evaluation item, in which the 
student is asked how well he feels he is doing. Answer a 
(e.g, “I think I am doing well and would like to finish 
now”) leads him directly to the end of the lesson. An- 
swer b (“I’m doing fine, but I would like some review” ) 
takes him to item 8 for a brief review of the materials. 
Answer c (“I don’t understand this lesson at all”) takes 
him back to the start of the entire lesson. 


EXPERIMENTS COMPLETED 


Following the development of the experimental teach- 
ing machine, research was started on a number of method 
variables, task variables, and student variables in auto- 
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mated teaching. The first study (Silberman et al., 1960) 
investigated the effects of one form of machine Tespon- 
siveness or branching, using over 400 items in logic. In 
this study the full capability of the automatic teaching 
machine was available. Thus a much more elaborate 
branching procedure was possible than that used in the 
previous studies with manual equipment. 

Tt was hypothesized that the performance of students 
receiving a branching sequence would be superior to that 
of students receiving a fixed sequence, with respect to 
Scores on a criterion post-test. In the branching group a 
student who had made errors on relatively difficult “main 
stream” items covering a topic was branched to a longer 
remedial list of simpler items covering the same topic. It 
was anticipated that by providing extra remedial items to 
students who had difficulty, the machine would minimize 
the number of items seen by the brighter student and the 
number of errors made by those with low aptitude. The 
over-all effect was expected to increase the amount of 
learning in a given time for each individual. 

Student aptitude and training time as well as instruc- 
tional materials were Statistically equated for the two 
groups. With this design, obtained differences could not 
be attributed to differences in student scholastic aptitude, 
to amount of study time, nor to the different characteris- 
tics of the materials that were presented to the two 
groups, but only to the responsiveness of the machine to 
student errors when branching to remedial material. 

Mastery of the topics covered by the training items was 
tested immediately after the training session. Analysis of 
the test scores yielded no significant difference between 
the two groups. Several factors indicated that this finding 
resulted from the particular method of branching used 
and from the inadequacy of the remedial items. Detailed 
analysis of item errors cued that the low aptitude stu- 
dents who received remedial materials did not benefit 
from them since they committed the same kinds of errors 
on the remedial materials as on main-stream items. The 
branching Procedure also permitted only òne trial 
through the items, so that even if the student still did 
not understand the topic after completing the remedial 
items he was sent on to the next topic. In addition, the 
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remedial items differed from the main-stream items only 
in the size of step or difficulty level, rather than in the 
approach used to cover the topic. 

Another experiment (Melaragno et al., 1960), this 
one using manual equipment, compared an optional 
branching procedure with the fixed sequence method. 
Whereas the previous study used branching which de- 
pended on student response errors, this study was 
designed to evaluate a procedure in which the student 
was allowed to branch at his own discretion. Three 
groups of high-school seniors were drawn from five 
schools. The first group received a set of teaching items 
in fixed sequence. Students in this group read each item 
in turn and were not permitted to view items already 
completed. Students in the second group received the 
same items as those in the first group but were permitted 
to back up one item at a time and review previous items 
that had already been covered. The third group received 
a five-page, single-spaced test consisting of the same ma- 
terials presented to the other two groups, except that 
teaching items were cast into statement form and organ- 
ized into paragraphs. Members of the third group were 
permitted to branch at their own option either to carlier 
material or ahead to material not yet covered. Mastery 
of the topics was tested immediately after the training 
session. The analysis of test scores showed a significant 
difference between the optional branching group and the 
fixed sequence group in favor of branching. 


EXPERIMENTS PLANNED 


Several experiments are now being planned for 1961. 
The logic items which were written for the branching- 
fixed-sequence study are now being revised and the pro- 
gram structure extensively modified for another experi- 
mental attack on this problem. The branching decisions 
will be made not only on the basis of errors but also on 
the basis of student response time and self-evaluation. 
If a student takes too much time on certain topics or 
feels that he needs some review, extra remedial items 
will be presented to him. A much greater variety of ap- 
Proaches will be used in the remedial items and students 
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will be halted until they have satisfactorily learned each 
topic. 

nother study being planned will not use the com- 
puter-controlled machine. One group of students will be 
required to give overt responses to instructional items 
presented manually. Students in a second group will re- 
ceive the same materials, but cast in the form of state- 
ments; members of this group will not be required to 
give an overt response nor permitted to branch among 
items. A third group will receive the same materials and 
conditions as the second group except that branching 
will be encouraged. Learning will be compared for the 
three conditions. This design is expected to provide some 
clarification of recent conflicting evidence concerning the 
relative effectiveness of these methods. 

SDC is planning a third project to develop an exten- 
sive sequence of items in high school mathematics using 
the computer-controlled teaching machine. These items 
will be used in an experiment in a number of high 
schools to answer certain practical questions concerning 
the application of auto-instructional methods in the pub- 
lic schools. The study will not be primarily concerned 
with the general question of whether auto-instructional 
methods teach, but with a determination of the kinds 
of students and the kinds of materials for which these 
methods are most effective. It will use two types of mate- 
tials: rote and conceptual. The rote material will require 
specific definitions and memorization of factual detail. 
The conceptual material will require a progression of 
thought leading to principles which generalize to novel 
problem situations. Three kinds of students will be used: 
over-achievers, normal-achievers, and under-achievers. 
The results of this study should yield valuable informa- 
tion on methods of applying different self-instructional 
materials to students of different motivational character- 
istics. It is expected that when conventional teaching 
methods are used, over-achievers will perform better than 
under-achievers on rote materials which have little in- 
trinsic appeal. On the more interesting conceptual topics, 
little difference is expected between the two groups. 
When self-instructional materials are used to supplement 
the regular instruction, it is expected that performance 
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for under-achievers and over-achievers will be essentially 
equated as the result of elimination of motivational dif- 
ferences between the two groups. This equating of per- 
formance is expected for both rote and conceptual mate- 
tials. The net effect of auto-instruction is expected to be 
greater for under-achievers than for over-achievers on 
tote materials, but no different for the two groups on 
conceptual material. i 


FUTURE DEVELOPMENTS 


The computer-controlled teaching machine presently 
used at SDC can teach only one student at a time. Plans 
are now being developed for an expanded facility, using 
a larger computer control unit, that will give automated 
instruction to 20 students simultaneously. The same facil- 
ity will provide automatic storage, analysis, and printout 
of administrative and counseling data normally associated 
with an operational school system. Development of the 
facility will permit the study of important interactions 
among students, teachers and school administrators. 
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Teaching Machines and Individual 
Differences* 


Arruur R. JENSEN 
University of California 


In science it is not unusual for a new method or ap- 
paratus or technique to give rise to new areas of basic 
tesearch. At times technical innovations will uncover a 
tealm of phenomena never before accessible to scien 
investigation. But, more often a technical developmen 
will merely force into greater prominence an already 
known but perhaps long neglected realm of phenomena. 
The scientific study of this realm then becomes a neces- 
sity if the technical innovation is to realize its full poten- 
tial in scientific research and ultimately in its application 
to practical problems. k t 

Sich is the case with teaching machines. The recen 
upsurge of this innovation in educational technology 
forces us to confront a number of fundamental psycho- 
educational problems. Of course, the research that first 
Comes to mind with respect to teaching machines is 
bound to have a practical and technical orientation, deal- 
ing with problems of design, programming, the compara- 
tive efficiency of teaching various subjects, and so forth. 
Not far behind these more immediate practical considera- 
tions will be some concern with many of the facts, con- 

6 
* This article originally appeared in Automated Teaching 


Bulletin, 1960, 1, 12-17, and is reprinted here with the per- 
mission of the author and the editor. 
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cepts, and theories derived from the experimental psy- 
chology of human learning. Thus, teaching machines are 
of interest not only to the applied psychologist, but also 
to the psychologist devoted to so-called “pure” or “basic” 
research on human learning. Most of the interest in 
teaching machines in academic circles today has been 
evinced either by experimental psychologists or by educa- 
tional psychologists who seem to be more interested in 
the immediate practical applications of teaching ma- 
chines. So far it appears that differential psychologists, 
i.e., those concerned with the study of individual differ- 
ences, have not shown an active interest in teaching ma- 
chines. 

Perhaps because of certain historical accidents in the 
development of scientific psychology, such as the fact 
that experimental psychology very early took the physical 
sciences as its model, the experimental study of human 
learning and the study of individual differences have 
developed quite separately. These two fields—experimen- 
tal psychology and differential psychology—traditionally 
have left each other pretty much alone. As a result, there 
is a large and rather clearly defined gap in our psychologi- 
cal knowledge of human behavior. This strange and 
bewildering gulf in psychology, which continues to per- 
sist, has been a detriment to our efforts to gain a greater 
understanding of human behavior. Now it appears that 
research on teaching machines may ultimately force a 
rapprochement between the experimental psychology of 
learning and the study of individual differences in learn- 
ing. 

The presence of individual differences in school per- 
formance are so obviously great as to have become tradi- 
tionally one of the primary concerns of educational 
psychology. But nearly all of the efforts to measure indi- 
vidual differences have been what might be called “static.” 
Rather than studying individual differences in the dy- 
namics of behavioral change, we have studied only the 
end products of learning by means of our intelligence 
tests, aptitude tests, and achievement tests. How people 
differ in the processes by means of which behavioral 
change takes place is truly an unknown and unexplored 
territory in psychology. 


220 MACHINES AND InprvipvAL DIFFERENCES 


Individual differences in learning will not be overcome 
or will not suddenly become unimportant when teaching 
machines are used. In all probability individual differ- 
ences will be accentuated. The teaching machine, far 
more effectively than the human teacher, can capitalize 
on individual differences if they are properly taken into 
account. I am not referring here only to such gross and 
obvious individual differences as time or number of trials 
needed to learn a given quantity of material. There are 
undoubtedly many othet more complex ways in which 
people differ in learning. For example, there are learning 
curves for two persons which are almost identical under 
conditions of distributed practice, but which diverge 
greatly from each other under conditions of massed prac- 
tice. Among more or less equally intelligent college stu- 
dents one can observe great individual differences in ma- 
chine learning under conditions of subject-pacing and of 
machine-pacing. Some students seem to learn more easily 
and quickly when they work at their own pace, while 
others do better when the pace is set by the machine. 
Our usual intelligence tests and aptitude tests will be of 
little value in discovering the kinds of individual differ- 
ences that will increase our understanding of performance 
on teaching machines. The kinds of learning difficulties 
that will show up in working with teaching machines will 
not lend themselves to diagnosis by means of the pres- 
ently available psychological tests, At present we do not 
even know what we should measure in order to get a bet- 
ter understanding of how and why various persons differ 
in their performance on teaching machines. It seems safe 
to say that individual differences in performance on 
teaching machines cannot be acounted for merely in 
terms of a simple unidimensional concept of “intelli- 
gence” or “learning ability.” Therefore, in order really 
to understand performance on teaching machines, so that 
the diagnosis of learning difficulties can be effective and 
so that in the use of teaching machines subject variables 
aes individual differences) as well as independent vari- 
ables (i.e, those controlled by the machiné) can be 
taken into account to achieve maximum efficiency, we 


must know a great deal about individual differences in 
learning. : 
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the need for a long-term program of systematic research 
on individual differences in human learning. This pro- 
gram is already underway in the writer’s laboratory. 

The first requirements of such a project are of a tech- 
nological and methodological nature. First of all, a highly 
standardized experimental procedure for investigating 
learning variables and measuring performance is needed. 
For this purpose we are using a specially constructed 
apparatus which has many points in common with teach- 
ing machines. In some ways it is more limited and yet 
more flexible than a teaching machine need be. 

It is essentially three parts: (1) a stimulus display 
unit, consisting of a screen onto which various stimuli 
can be projected, (2) a response unit, consisting of a 
panel of buttons which can be varied in number and po- 
sition, and (3) a control unit, which consists of (a) a 
“teader” for the punched tape on which the entire ex- 
periment is programmed, (b) a bank of electrical coun- 
ters which tabulate the subject’s performance throughout 
the course of learning, and (c) “criterion counters” 
which stop the machine when the subject has attained 


pject-paced or machine-paced. } 
The apparatus permits trial and error selective learning 
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experiments on a very simple level in terms of the con- 
tent of the leaming, since we wish to avoid as much as 
possible the effects of previous learning which may show 
up in the form of associations, coding tricks, mnemonic 
devices, etc. For the same reason, we are using non- 
verbal stimuli. Students are learning to associate pressing 
certain buttons in response to such stimuli as squares, 
circles, and triangles colored red, blue, and yellow. Since 
at present we are interested only in simple associative 
learning, the use of such simple, easily discriminable 
stimuli obviates the need for the student to learn the 
stimuli themselves as would be the case if we used non- 
sense syllables. Eventually, of course, it will become de- 
sirable to use more complex stimuli and to introduce 
verbal materials. One advantage of the present setup is 
that it permits the experimenter to perform a great va- 
riety of learning experiments all of which still involve 
essentially the same experimental parameters: He can 
change one variable at a time while all others are he 
constant. Thus a continuity and integration of findings 
from many different experiments will be possible: This is 
3 oe important feature as our research is envisaged as 
very long-term program. ; 
What is the Tee plan or “ hilosophy” of this 
research program? It may be considered an attempt to 
discover the basic dimensions underlying individual dif- 
ferences in various learning phenomen, much as differ- 
ential psychologists have sought to discover the dimen- 
sions underlying cognitive abilities or personality charac- 
teristics. Since learning depends upon the nervous Sys- 
tem, a product of biological evolution, 
assume that there is a limit 
which people differ in the learning process- t i 
should be a limited number of fundamental dimensions 
in terms of which individual differences in various kinds 
of learning may be described. It seems most unlikely Ha 
there should be no “structure” to learning ability 1n the 
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ing from Clark Hull, offers us some clues as to the possi- 
ble nature of the dimensions we might discover. For 
example, individual differences in performance on pe 
ing tasks might be thought of as a joint effect of individ- 
ual variations in such Hullian variables as habit strength 
(ease of formation of associations), drive strength, in- 
hibition (a function of effortfulness of response, repeti- 
tion, and non-reinforcement), and so on. For example, 
differences in habit strength may show up in terms of 
sheer speed of learning when subjects are equated on 
other variables. Differences in inhibition may show up in 
rate of extinction, in differential rates of learning under 
massed and spaced practice, and in the magnitude of 
reminiscence effects. At present we have no idea how 
many basic dimensions we will ultimately find. One may 
hypothesize, for example, a dimension of susceptibility 
to “interference” or response competition which would 
be expected to show up in experiments on proactive and 
tetroactive inhibition and associative interference. On 
the other hand, individual differences in these “interfer- 
ence” phenomena may be derived from more fundamen- 
tal dimensions, perhaps, for example, from an interaction 
between habit strength and drive. These are the kinds 
of problems we are setting out to study. 

Another ponien on which sooner or later large-scale 
tesearch will have to be launched concerns the relation 
between acquisition and retention. It may be that the 
dimensions we discover underlying individual differences 
in learning or acquisition will not be the same as those 


involved in the long-term retention of what has been 
learned. 
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interference. It is necessary to determine iabili 
of measurement of individual differences Seon 
ee This usually involves testing the same ais 
a sg ain on equivalent forms of the learning task. 
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laboratory experiments could later be reliably measured 
in individuals and could be identified in theit perform- 
ance under the less simple and less rigorous conditions 
approximating those found in the practical use of teach- 
ee raeniaes, First in this wilderness we must seek the 
pase dimensions in the most simple an “purified” 
orms of various learning phenomena. What is discov- 
ered and proved in the laboratory may later be demon- 
strated in more gross practical situations. 

Still quite far in the future in regard to the work 
planned for the writer’s laboratory is a third stage of this 
Tesearch. It will consist of relating the dimensions © 
learning to the factors already discovered in the realm of 
intellectual or cognitive abilities and in the personality 
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forms of school learning are being sought, so that the 
work of our laboratory is not completely divorced from 
practical problems of learning. 

It is hoped that eventually in the not too distant fu- 
ture a truly scientific and practically fruitful rapproach- 
ment will be achieved between learning theory, the ex- 
perimental psychology of human learning, differential 
psychology, the psychology of school subjects, and the 
applied technology of teaćhing machines. 
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Programmed Self-Instruction: 
Possibilities and Limitations* 


Gene C. Fusco 


School Administration Branch, 
U.S. Office of Education 


POSSIBILITIES 


self-instruction maintain 


Proponents of programmed 
d of teaching; the basic 


that this is not a new metho 
procedure is found as far back as the Socratic dialogue, 

hat is new, they say, is the development of a science 
and a technology based on a method employed by the 
great teachers of the past. Thus, some aspects of the art 
of teaching may become a part of the science of learning 
through programmed self-instruction. 

Theoretically, the mechanical tut 
saving devices for the teacher since th 
techniques are tailored to the learning speed of the indi- 
vidual student. This method provides the student with 
as much practice as he needs, permits the rapid learner 
to cover the material quickly and the slower student to 
proceed at his own pace. As a consequence, the classroom 
teacher may be freed from the burdensome and time- 
consuming tasks of presenting material and taking pre- 


ors represent labor- 
e self-instructional 


a 
* This article is an abridgement of a paper which originally 
appeared in The High School Journal, 1960, 59, 85-90. It is 
used here with the permission of the author and the editor. 
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cious class time to repeat material for students who didn’t 
get it the first time. ; 

Advocates of  self-instructional techniques further 
maintain that this method may be adapted to home 
study and can save the teacher from devoting time to 
the correction of individual assignments. Self-correcting 
homework materials, it may be noted, have been pre- 
pared in high school English by Educational Testing 
Service (Diederich, 1960). Self-instructional materials 
may also be used by temporarily homebound students 
who may keep up with assignments while unable to at- 
tend regularly scheduled classes. In addition, some pro- 
Eos that programmed materials may be used in small 

igh schools with limited curriculum offerings to meet 
the needs of students who request courses not included 
in the school program. 7 

Students working with self-instructional materials will 
be enabled to work privately and independently and, as a 
consequence, recognize the fruits of their own labor. It 
is contended that this will result in a lessening of in- 
vidious comparisons of their achievements with that of 
their classmates. 

According to Simon Ramo, in the high school of the 
future there will still be human teachers and classroom 
sessions, but each student will spend a large proportion 
of each day interacting with computer-controlled teach- 
ing machines. The machines will tutor the student, 
schedule the curriculum to suit his individual abilities, 
and provide his human teachers with a detailed descrip- 
tion of his academic strengths and weaknesses. Ramo 
foresees the need for highly skilled teachers who can work 
closely with subject matter specialists and “teaching 
engineers” in modifying and Tevising the teaching de- 
vices and programs (Ramo, 1957) 


LIMITATIONS 


There are now dozens of different types of devices that 
could be classified as teaching machines, most of which 
Tepresent minor variations of two, or at best, three or 
four basic designs. At present, good problem materials 
to be used with teaching machines, and procedures and 
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principles for the preparation of such materials have not 
kept pace with the development of the devices them- 
selves. There is a lack of extensive empirical data on the 
effectiveness of this method of teaching. Despite some 
favorable findings of research in self-instructional tech- 
niques, there is little evidence on the factors that make 
that particular machine design or set of instructional 
items effective. There is much to be learned about the 
preparation of good items, and there are hardly any cri- 
teria to determine whether the items in a program are 

“good.” 

Another limitation is that the selection of a particular 
teaching machine places severe limitations on the type 
of variables the researcher can investigate. Teaching ma- 
chines, for the most part, are single-purpose devices with 
limited adaptability. Thus, it is possible for the machine 
to dictate the direction of research, whereas, ideally, re- 
search considerations should direct the development of 
the machine. A <a 

Another problem in reference to the inflexibility of 
programmed self-instructional methods is that a particu- 
lar program may not be optimally effective for all mem- 
bers of a typical class. Students, after all, differ in both 
experience and in inherent ability, or “intelligence.” An 
“ideal” sequence of items for one student may be less 
effective for some other student. 5 F 

Teaching machines present items in an essentially 
predetermined sequence, permit the student to respond, 
and give him immediate feedback. But does the fact that 
a student is shown the correct answer, or selects the cor- 
rect answer after several attempts, insure that he has 
actually acquired knowledge? The machine does not 
“know” whether the student has mastered the material 
the way a human instructor may know through observa- 
tion and questioning. Lok 

Professor Skinner, for example, insists that a student 
through machine instruction will be enabled to proceed 
at his own level, advancing as rapidly as he can. Under 
these conditions, he says, the conventional grading sys- 
tem will have to be overhauled since a grade will be 
useful only in showing how far a student has gone. Thus, 
a “C” might mean that he is halfway through a course. 
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Given enough time, says Skinner, he can achieve an SAR 

In challenging Skinner's point of view, one psycholo- 
gist calls attention to the question of transfer and cites 
studies on overlearning which tend to show that pet- 
formance may not mirror how much has been learned. 
He questions whether the completion of a programmed 
course is the final criterion of learning (Kendlar, 1959). 

Other limitations of programmed instruction include 
the large expenditure of money, the great amount of 
time, and the application of special skills necessary to 
produce a program; the difficulty of determining the 
specific body of knowledge to be programmed; and the 
fact that the method lacks a philosophy of application 1n 
the instructional process. 
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Appendix A 


Teaching Machine Terms: 
A Glossary* 


Dersmonp L. Coox 
Purdue University 


In compiling this glossary, the author drew from sev- 
eral sources in addition to his own experience and ac- 
quaintance with the area of “teaching machines.” The 
primary references were Eugene Galanter, Automatic 
Teaching: The State of the Art (John Wiley, 1959); 
William Carr, Self-Instructional Devices: A Review of 
Current Concepts, Wright Air Development Center, 
Technical Report 59-503, August 1959; and Edward Fry, 
Glenn Bryan, and Joseph Rigney, “Teaching Machines: 
An Annotated Bibliography,” AV Communication Re- 
view, Vol. 8, No. 2 (Supplement 1), 1960. A 

Unpublished papers and reports, personal correspond- 

ence, and comments received on the first draft from per- 
sons actively engaged in research in the area were a 2 
utilized, Some terms were secured from a similar draft 
glossary prepared by Charles Darby, Department of Psy- 
chology, Purdue University. 
Augmenti in which guidance through a program 
is EE or insoducing bits of information 
that will lead the student into a new concept. A 
Auto-instructional methods. A comprehensive term suggeste 
+ Thi se iginall eared in Audiovisual Instruc- 
E used here with permission of 
the author and the publisher. A 
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by Lunsdaine and Klaus to describe instruction characterized 
by the controlled presentation of material, the elicitation of 
appropriate response, guidance with respect to the subject mat- 
ter, and control of the way in which learning proceeds. 
Automated instruction. Instructional methods considered gen- 
erally to comprehend any means, devices or material, whereby 
teacher or tutor functions (actual or desirable) are~ replaced, 
or provided, by a wholly or partially automated sequence of 
instructional segments that is prepared in advance and is capa- 
ble of instructing effectively when presented without direct 
intervention or modification by a teacher; Also called “auto- 
matic teaching.” 

Automatic teaching. See “Automated instruction.” 


Automatic tutoring. An individual instructorless method of 
teaching which is an automation of the classical process of indi- 
vidual tutoring. 

Choice. Refers to the selection of an answer from several al- 
ternatives presented to the subject as opposed to having the 
subject construct or write out an answer, 

Construction. The process of Tequiring the subject to write out 
or prepare an answer as opposed to choosing one of several 
alternative answers, 

Confirming mechanism. A device or means by which the stu- 
dent has his response confirmed as being right or wrong. Cor- 
rectness or error can be indicated through the presentation of 
visual or auditory signals, or, in the case of constructive an- 
swers, the presentation of the correct or model answers. 


Conversational chaining. A term introduced by Barlow to de- 
scribe a program wherein the movement from item to item 
follows less the question-answer Pairs in other programs and 
more the natural train of conversation so that the lesson be- 
comes the main unit of concern for the student rather than the 
item. 

Cue, vanishing. A prompt presented originally, but gradually 
reduced or eliminated, thereby Tequiring the student to provide 
the responses independently. 

Display mechanism. The mechanical means by which the pro- 


grammed set of materials is Presented visually or aurally to the 
subject. 


Error. An incorrect or non-appropriate answer to a specific item 
in the program. 

Error rate. Refers to the number or Percentage of a given group 
of subjects incorrectly responding to a specific item on the pro- 
gram. A high degree of error would probably indicate a need 
for revision in the program, 
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Fading. The gradual withdrawal of stimulus support in pre- 
senting items. 

Feedback. Communicating to the subject pursuing a sequence 
of programmed materials the information needed to modify 
responses so that failures or errors can be eliminated and correct 
responses maintained. 

Frame. A single item or statement is exposed at a time. The 
exposed material constitutes a single frame. 

Frame, forced. A stimulus frame presented to the student forc- 
ing him to respond correctly by obvious nature of the answer. 
Frame, response. That part of a teaching machine or pro- 
grammed textbook which permits the subject to record his 
response to the item presented by means of the stimulus frame. 
(See stimulus frame. ) 

Frame, stimulus. That part of a mechanical teaching machine 
or programmed textbook containing the individual items, in 
the form of a question or statement, to which the student is 
to respond. Also refers to the mechanical part of the teaching 
machine which allows the examinee to see the item. 

Hints. Devices used to direct the students’ behavior in the 
desired direction. Used to increase the likelihood of a correct 
response. 

Item, augmenting. An item supplying new information but not 
requiring the student to make a relevant response. 

Item, delayed review. An item which allows for the distribution 
of practice. Differs from other items only in terms of presenta- 
tion. 


Item, dovetailing. An item req 
rate responses to separate stimul 
confused. 

Item, fading. An item requiring the student to review what 
has been presented. In addition the item withdraws informa- 
tion successively. Similar to Skinner’s “vanishing technique. 


T f 3 ni 
Item, generalizing. An item presenting a verbal statement poin 
ing to a common characteristic of several specific problems 
already presented to the subject. pw 
i i i i t to review 
Item, interlocking. An item that requires a studen! 
the established skills while new information 1s being presented. 
d i i iri information or Te- 
Item, lead ir. An item not requiring new mM 0 
hearsal of old skills but functions to orient the subject to a 
problem and prepare him for new information. 
Item, restated review. An item requiring a rehearsal of the skill 
where a problem is restated. 


uiring the student to make sepa- 
li which otherwise may become 
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Item, rote review. An item presenting a problem identical to 
one presented earlier. 

Item, specifying. An item which exemplifies a general rule or 
principle. 

Item, subject matter. An item classified with respect to its 
subject matter content. 


Item, tab. A specialized term referring to having the subject 
pull a tab to indicate his response rather than writing out 
an answer or selecting a choice. 

Leading. The student is first asked to talk about familiar things 
using his everyday vocabulary. He is then led to discuss relations 
among these. Technical terms are then slowly inserted. 
Learning machine. A term often applied to teaching machines. 
An inappropriate term for describing a mechanical teaching 
device since it implies that a machine learns. 

Matching. Procedure used in some Skinner machines to inform 
student of correctness of his response. After writing his re- 
sponse, the student moves a lever which exposes the correct 
answer with which the student compares or “matches” his 
response. 

Operant behavior, Behavior which Operates or acts upon the 


environment. A fundamental concept in Skinnerian learning 
theory. 


Pace. The rate at which the subject is permitted to work 
through the programmed material. 


Pacing, controlled rate. Control of the subjects’ rate of re- 


sponding by features of the mechanical device utilized to present 
the program, 


Pacing, self. The rate at which the subject might complete 


the material at his own rate depending upon success on the 
previous steps. 


Panel. A short passage of Prose material, graphs, and similar 
material which are presented or studied along with the discs 
in the Skinner device. 


Paper teaching machine. Refers to the “scramble 
grammed” textbook type of self-instructional devices. 


Porter device. A device similar to the Skinner disc machine, 
except instead of using round discs, a regular sheet of paper 
is inserted into the mechanism. The Paper contains the pro- 
grammed material and spaces for insertion of answers, 


Pressey device. The earliest known device (1926) originally 
developed for use with multiple choice tests. Device could be 


or “pro- 


| 


ð 
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set so that items missed could be skipped or repeated until 
success was established, a raw score obtained, and an item 
analysis or error count secured. 

Program. The subject matter that is to be learned by the stu- 
dent via the machine or other device. 

Programmed book. A special book in which the subject matter 
to be learned has been arranged into a series of sequential 
steps leading from familiar concepts to new materials. Differs 
from a “scrambled textbook” in that the content is arranged 
so that the student proceeds directly from one step to the next, 
or one succeeding page to the next, rather than skipping 
around. The student generally is asked to construct a response 
as opposed to choice. 

Programmed learning. A term sometimes used synonymously 
to refer to the broader concept of “‘auto-instructional methods. 
Programmer. Generally, a curriculum specialist who subdivides 
the material to be learned into the sequential steps for later 
use with the mechanical method of presenting the program. 
Programming. The process of arranging the material to be 
learned into a series of sequential steps; usually moves the 
student from a familiar background into a complex and new 
set of concepts, principles, and understandings. 

Programming, intrinsic. A method of programming materials 
that directs the erring subject along certain corrective pate 
before he is permitted to proceed to the next step Pai tt S 
program. Requires that each step contain multiple choi 
answers. 


Prompt. Some type of verbal or symbolic cue which 
the desired response from the subject. 


Prompting. The method or sequence 0! 

symbolic cues to encourage responses. 

symbolic, or auditory. 

Reinforcement, immediate. The process of provi 

ject with immediate feedback or information 

success or failure of his performance. ‘ 
ard for responding 


Reinforcement mechanism. Some type of reward for oe 
correctly to the items in the display. à motivational Ea, 
causing the individual to keep working at ne si confirming 
Sometimes considered as an integral part © 


mechanism. i 

X . . : ich contains 
Reinforcer, immediate. A self-instructional aid Mn eiie 
a built-in system of providing the student E 
knowledge of the success or failure of his perto: 


facilitates 


f providing verbal and 
Ga be visual, verbal, 


iding the sub- 
regarding the 
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Response device. A type of teaching aid which permits the 
student to practice certain types of responses. No stimulus is 
presented as part of this type of device. 

Response mechanism. A device which permits the student to 
record his response to the items presented in the display 
mechanism. Usually is either the selection of one of several 
choices or writing an answer to the item in the program. 
Response mode. The form of the Tesponse a student makes 
while working on a program. 

Scrambled book. A special book containing material to be 
learned in programmed sequential form, but in which the 
student is directed to different pages not necessarily in con- 
secutive order. By means of alternate choice tesponses at each 
step, branching to new or review material is made possible. 
Self-instructional device. A mechanical or paper device which 
presents a set of planned sequential materials to be learned 
and which the student can complete in the absence of a live 
instructor and at his own rate of speed. 


Shaping. The building of a behavior or set of behaviors 


through the differential teinforcement of progressively more 
adequate forms of behavior. (Skinner) 


Skinner device. A mechanical d 
programmed materials. At each 
an answer and evaluate its co: 
before proceeding further in th 
the forerunner of later model “teaching machines,” 


Skinner disc, A round, flat, record-like device which contains a 
set or series of program materials for the Skinner device. Con- 
tains the questions to be answered, spaces for recording stud- 
ent’s answers and the correct response, as well as for making a 
record of successful or unsuccessful performance. 


Step. The increment in subject matter level to be learned with 
each succeeding item or frame in the program. 


levice which presents a set of 
step the subject must construct 
rrectness with a model answer 
e program. Generally considered 


step would indicate a 
rogram. 

; pe of teaching aid which presents mate- 
tials to the subject through one or several of the various senses. 
No active Tesponse is required on the part of the subject during 
the presentation of the materials. 


Stimulus-response device. A teaching aid which not only pre- 


sents material to the subject through any or several of the 
various senses, but also Tequires a response to the stimulus pre- 


telatively large number of frames in the pı 
Stimulus device. A ty) 
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sented in order to progress further in the program of instruc- 
tion. 

Terminal behavior. The behavior a program is designed to 
produce. 

Track, multiple. A provision within the programmed material 
for allowing subjects to pursue alternative subdivisions of the 
program in terms of their successes or failures with earlier sec- 
tions of the sequence. 

Track, single. A common set of programmed materials which 
all subjects work through, there being no alternative program 
such as in the multiple track situation. 

Vanishing. The basic problem here is to evoke a given bit of 
behavior, at least once, in the presence of an appropriate stimu- 
lus, so that it can be reinforced. One solution is to begin with 
stimuli which already control the behavior and to reduce them 
slowly as learning proceeds. 


Appendix B 


1. Additional Sources of Information 
on Programmed Instruction 


BOOKS 


Gatanter, Eucene (Ed.). Automatic Teaching: The 
State of the Art. New York: Wiley, 1959. 

Lumspaing, A. A., and Craser, ROBERT (Eds.). Teach- 
ing Machines and Programmed Learning: A Source 


Book. Washington, D.C.: Natl. Educ. Assn., 1960 
(724 pp.). 


PERIODICALS CARRYING RELEVANT 
ARTICLES REGULARLY 


AID, Institute of International Research and Development 
ducational and Training Methods Division 
Box 4456, Lubbock, Texas 
Audio-Visual Communication Review, Department of Au- 
diovisual 
Instruction, National Education Association 
1201 Sixteenth Street, N.W, 
Washington 6, D.C. 


Contemporary P. sychology, American Psychological Associ- 
ation 


1313 Sixteenth Street 


Washington 6, D.C. 
Programed Instruction, The Center for Programed Instruc- 
tion, Inc. 
365 West End Avenue 
New York 24, New York 


The Automated Teaching Bulletin, Rheem Califone Cot- 
poration 


1020 North LaBrea Avenue 
s Angeles 38, California 
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PERIODICALS CARRYING RELEVANT 
ARTICLES IRREGULARLY* 


Audiovisual Instruction, Department of Audiovisual In- 
struction 
National Education Association 
1201 Sixteenth Street, N.W. 
Washington 6, D.C. 
Harvard Educational Review, Harvard University 
Cambridge 38, Massachusetts 7 
Journal of Educational Psychology, The American Psycho- 
logical Association, Inc., 1333 Sixteenth Street, N.W. 
Washington 6, D.C. 
Journal of Experimental Analysis of Behavior, Psychology 
Department 
Indiana University 
Bloomington, Indiana 
Journal of Psychology, The Journal Press 
2 Commercial Street 
Provincetown, Massachusetts 
Phi Delta Kappan, Eighth and Union Ayenue, 
Bloomington, Indiana Oats 
Psychological Reports, Southern Universities Press 
Box 1441, 
Missoula, Montana 


2. Manufacturers of Teaching Machines* 
Astra Corporation, 31 Church Street, New London, Con- 


necticut ; 
Deveraux Teaching Aids, Smith-Harrison, Inc. Box 717 
717 Devon, Pennsylvania a i : 
Dyna-Slide SESE 600 South Michigan, Chicago 5, Mli 


nois 1 h s 
Foringer and Co., Inc., 312 Maple Drive, Rockville, Mary: 
land 
General Atronics Corporation, 1 Bala Avenue, Bala-Cynwyd, 
Pennsylvania ai, 
Hamilton Research Associates, Inc., Box 38, New Hartfor 


New York ike 
Hughes Aircraft Company, Industrial Systems e Bo 

90904 Airport Station, Los Angeles ee 3, New 
Koncept-O-Graph Corporation, Box 533, 

York h, Chi- 
Manägement Research Associates, 185 North Wabas 

cago, Illinois 


* A partial list. 
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Programmed Teaching Aids, Inc., 3810 South Four Mile 
Run Drive, Arlington, Virginia 

Rheem Califone Corporation, 1020 North LaBrea Avenue, 
Los Angeles 38, California 

Western Design, U.S. Industries, Inc., Santa Barbara, Cali- 
fornia 


3. Sources of Programs* 


Doubleday and Company, Inc., 501 Franklin Avenue, Gar- 
den City, New York, Tutor Texts 

Harcourt, Brace and World, Inc., 730 Third Avenue, New 
York 17, New York 

Institute of International Research and Development, Educa- 
tional and Training Methods Division, Box 4456, Lub- 
bock, Texas 

McGraw-Hill Book Company, Inc., College Department, 330 
West Forty-Second Street, New York 36, New York 

Programmed Learning Materials, Encyclopaedia Britannica 
Films, Inc., 1150 Wilmette Avenue, Wilmette, Illinois 

Science Research Associates, Inc., 259 East Erie Street, Chi- 
cago 11, Illinois 

Student Self-Instruction Project, Earlham College, Rich- 
mond, Indiana 

TMI-Grolier, 235 San Pedro Drive, N.E., Albuquerque, New 
Mexico 

The Center for Programmed Instruction, 365 West End 
Avenue, New York 24, New York 


* A partial list. 
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